Product
The routing layer between your app and every model you use
Four things run on every request: a routing decision, a cache lookup, a health check and a cost record. Here is how each one works.
A policy is a small JSON object
No DSL, no console-only config. Version it with your code, attach it to a key, change it without a deploy.
{
"policy": "cost_optimized",
"quality_floor": 0.92,
"max_latency_ms": 4000,
"escalate_on": ["low_confidence", "tool_error"],
"tiers": {
"small": ["gemini-flash-lite", "llama-3-8b"],
"mid": ["gemini-flash", "claude-haiku"],
"frontier": ["gpt-5", "claude-opus"]
},
"cache": { "enabled": true, "similarity": 0.97, "ttl": "24h" }
}Routing policies
You set the bar. Thalera finds the cheapest way to clear it.
A policy is a quality floor, a latency budget and a set of model tiers. Every request is scored against that policy before a single token is generated.
Policies attach to an API key, a route or a header, so one service can run cost-optimized traffic while another runs latency-first — same endpoint, different behaviour.
- Cost-optimized: cheapest model above the quality floor
- Latency-first: fastest healthy model within budget
- Quality-first: frontier tier, cheaper models only for cache hits
- Pinned: a specific model, with failover still active
Semantic cache
Stop paying twice for the same answer
Real traffic repeats itself. Support questions, product lookups, boilerplate summaries — the same intent arrives thousands of times a day with slightly different wording.
Thalera embeds each prompt and serves a stored response when similarity clears your threshold. Cache hits cost nothing and return in milliseconds.
- Per-route similarity threshold and TTL
- Namespace isolation per tenant or user
- Explicit bypass header for must-be-fresh calls
- Hit rate and saved spend reported per route
Failover & load balancing
A provider incident should not be your incident
Thalera tracks health, error rate and rolling latency for every model it can reach. When one degrades, traffic shifts to an equivalent before your error rate moves.
Retries are budget-aware: a failed cheap call retries cheap, and only escalates when the policy says the answer is worth more.
- Mid-stream re-route on timeout or 429
- Weighted balancing across regions and providers
- Per-key concurrency and spend ceilings
- Graceful degradation instead of hard failure
Observability
Know the cost of a feature, not just of a month
Every call is logged with the model chosen, why it was chosen, tokens in and out, cache status, latency and exact cost.
Slice by key, route, team, customer or model. Export to your warehouse, or alert when a route drifts above its budget.
- Live cost and latency dashboards
- Per-request routing decision trace
- Budget alerts and hard spend caps
- OpenTelemetry export and raw log streaming