Caching — Docs

Caching

You never pay twice for the same question. Celon looks each request up in a dual cache in Exact → Semantic order and, on a hit, responds immediately without an upstream call. No failure in the cache layer ever fails a request — it only ever degrades to a miss.

Exact cache

The fields that affect the completion (model, messages, temperature, top_p, max_tokens, response_format) are serialized into stable, key-sorted JSON, then the full response is stored under a SHA-256 hash of it as the key. An identical request hits instantly and is marked cache_hit: "exact" in the response metadata.

Semantic cache

Catches questions that mean the same thing even when worded a little differently. It embeds the first 500 characters of the system prompt plus the last user message with openai/text-embedding-3-small, compares cosine similarity against an index of recent requests, and returns the stored response when it clears the threshold (cache_hit: "semantic").

Matching only happens within the same model value — each model has its own independent index.
The index is a ring buffer of the most recent 512 entries per model, and when Upstash is configured it is mirrored to KV so it survives restarts.
Embedding uses the OpenAI adapter, so without an OPENAI_API_KEY the semantic layer disables itself automatically (the exact cache keeps working).

The threshold is tuned with the SEMANTIC_CACHE_THRESHOLD environment variable and defaults to 0.90. It’s a value set from real measurements — paraphrase pairs that differ only in wording cluster at cosine similarity 0.92–0.95, while pairs that differ in meaning (the dangerous ones) fall to 0.68 or below, leaving a comfortable margin in between.

0.95 and up — conservative. Practically only requests with almost no wording difference are caught, and from 0.97 on it’s no different from the exact cache.
0.90 (default) — the balance point that catches paraphrases while safely filtering out pairs that differ in meaning.
Below 0.85 — risky. A cached response could go out for a question that means something different, so it’s not recommended.

Cache eligibility conditions

Only requests that satisfy all of the conditions below are eligible for cache lookup and storage.

Condition	Description
`stream !== true`	Streaming requests are not cached.
`temperature ≤ 0.3`	Only deterministic requests are cached. Since OpenAI’s default is 1.0, you must explicitly lower temperature to make a request eligible.
`no tools`	Function-calling requests are not cached.
`celon.no_cache !== true`	There must be no per-request opt-out.
`body < 32,000 chars`	The combined text of all messages must be under 32,000 characters.

TTL and backend

TTL is set with the CACHE_TTL_SECONDS environment variable and defaults to 86,400 seconds (24 hours).
The backend is Upstash Redis REST when UPSTASH_REDIS_REST_URL / UPSTASH_REDIS_REST_TOKEN are set, otherwise an in-memory LRU.
Cache writes happen asynchronously after the response is returned — they don’t add to request latency.

Zero-cost handling and the saved_usd calculation

A cache hit makes no upstream call, so it responds with cost_usd: 0, provider: "cache", and is recorded in the usage ledger at zero cost. The savings on a cache hit (saved_usd) are computed against the cost of the model that originally served the response — reuse a Tier 3 answer and the savings count only as much as the Tier 3 price, never assuming a frontier model was used. By contrast, routing savings that aren’t from the cache (a miss that dropped to a lower tier) compute saved_usd against the baseline anchor (anthropic/claude-fable-5).

saved_usd calculation

# cache hit — no upstream call
cost_usd  = 0
saved_usd = costOf(cached_model, prompt_tokens, completion_tokens)
# ↑ the cost of the model that originally served this response. Reusing a
#   Tier 3 answer counts only that much as savings (not a frontier baseline).

# cache miss, routed to Tier 2/3
saved_usd = max(0, costOf(baseline, tokens) - costOf(routed_model, tokens))

# Tier 1 · explicit model
saved_usd = 0