Documentation

Thalera speaks the OpenAI chat completions API. If your code already talks to OpenAI, it already talks to Thalera.

Quickstart

Create a key, then change the base URL your client points at. Nothing else in your application needs to change.

.env

# Before
OPENAI_BASE_URL=https://api.openai.com/v1

# After
OPENAI_BASE_URL=https://api.thalera.dev/v1
THALERA_API_KEY=thl_live_...

First request

Use the virtual model thalera-auto to let routing choose. You can also pass a real model id to pin it while keeping failover.

curl

curl https://api.thalera.dev/v1/chat/completions \
  -H "Authorization: Bearer $THALERA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thalera-auto",
    "messages": [
      { "role": "user", "content": "Summarize this ticket in one line." }
    ]
  }'

The response is a standard completion plus a thalera block describing the decision.

json

{
  "id": "cmpl_9xQ...",
  "model": "gemini-flash-lite",
  "choices": [{ "message": { "role": "assistant", "content": "..." } }],
  "thalera": {
    "policy": "cost_optimized",
    "reason": "low_complexity",
    "cache": "miss",
    "cost_usd": 0.000042,
    "latency_ms": 310
  }
}

Routing policies

A policy is attached to your key by default. Override it per request with headers when a specific call needs different behaviour.

Policy	Optimizes for
cost_optimized	Cheapest model above the quality floor
latency_first	Fastest healthy model within budget
quality_first	Frontier tier, cache still active
pinned	A named model, with failover

curl

# Per-request override
curl https://api.thalera.dev/v1/chat/completions \
  -H "Authorization: Bearer $THALERA_API_KEY" \
  -H "X-Thalera-Policy: latency_first" \
  -H "X-Thalera-Quality-Floor: 0.95" \
  -d '{ "model": "thalera-auto", "messages": [...] }'

Streaming

Streaming works exactly as upstream. If a provider fails before the first token, Thalera re-routes transparently; after the first token it fails over to a retry.

python

stream = client.chat.completions.create(
    model="thalera-auto",
    messages=[{"role": "user", "content": prompt}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

# Routing metadata arrives in the final chunk as
# chunk.thalera — including the model actually used.

Caching

The semantic cache is on by default for cost_optimized. Namespace it per tenant so users never see each other's answers.

python

client.chat.completions.create(
    model="thalera-auto",
    messages=messages,
    extra_headers={
        "X-Thalera-Cache": "on",        # on | off | refresh
        "X-Thalera-Cache-Namespace": tenant_id,
        "X-Thalera-Cache-TTL": "3600",
    },
)

Error codes

Status	Code	Meaning
400	invalid_request	Malformed body or unknown policy name.
401	invalid_key	Missing or revoked Thalera API key.
402	budget_exceeded	Spend cap for this key was reached.
409	no_eligible_model	No model met the quality floor and latency budget.
429	rate_limited	Per-key concurrency limit hit. Retry with backoff.
503	all_providers_degraded	Every candidate provider failed health checks.

Need something not covered here?

Batch endpoints, embeddings routing, self-hosted classifiers and private deployments are all supported.