Caching and What It Does to a RAG Bill
Caching is the highest-return optimisation available to a retrieval system, and it is also the one most likely to be implemented in the wrong place. Four distinct things can be cached — the prompt prefix, embeddings, retrieval results, and whole answers — and their returns differ by orders of magnitude, as does the risk each one introduces.
Do them in order of return divided by risk. That order is not the order people usually attempt.
The four caches
Prompt prefix caching. Providers can cache the processed form of a repeated prompt prefix, so the fixed part of your prompt is not charged at full input price on every request. As of this writing this is a standard capability and the mechanics vary by provider, so verify the terms yourself rather than assuming.
The return is large because the input term dominates a retrieval prompt, and a substantial part of it — the system instructions, the format rules, the few-shot examples — is byte-identical on every request. The requirement is that the stable part comes first, before the retrieved passages and the question. That is a prompt-ordering decision, which means the cost of taking advantage of this is roughly zero. There is no correctness risk: the cache holds your own bytes.
Query embedding caching. Store the vector for a query string you have seen before. Cheap to implement, and worth almost nothing, because embedding a short query was already a negligible share of the cost — as shown in self-hosting your embedding model. It does shave a round trip off latency, so treat it as a latency optimisation rather than a cost one.
Retrieval result caching. Store the retrieved chunk IDs for a query. Saves a vector search and a hop; the vector search was tens of milliseconds and a small cost. Modest return, one real risk: the cache must be invalidated when the index changes, or you serve results from a corpus state that no longer exists. Tie its lifetime to your ingestion cycle.
Answer caching. Serve a previously generated answer for a repeated question. Saves the entire cost of the request — the whole generation bill and nearly all the latency. Enormously the largest return, and the only one with a serious correctness risk, because a cached answer can outlive both the document it came from and the permission that allowed it.
| Cache | Cost saving | Latency saving | Risk |
|---|---|---|---|
| Prompt prefix | Large | Meaningful | None |
| Query embedding | Negligible | Small | None |
| Retrieval results | Small | Small | Stale results if not invalidated |
| Whole answers | Very large | Very large | Stale content; permission leakage |
Whether answer caching helps depends entirely on your query distribution
This is the question to answer before building anything, and it is answerable from a log.
Consumer-facing systems tend to have heavily repeated queries — a small number of questions account for a large share of traffic. Internal tools over a big corpus are often the opposite: highly varied, specific, rarely repeated. Answer caching is transformative in the first case and nearly useless in the second, and the difference is not something to guess at.
Measure it before you build it. Normalise a week of queries — lowercase, trim, collapse whitespace — and count how many are exact repeats. If exact repeats are a small fraction, an exact-match answer cache will not pay for itself. Do this before writing any caching code; it is an afternoon with a log.
Worked illustration, all inputs hypothetical. Suppose 20,000 answers a day at a hypothetical fully loaded $0.018 per answer — that is $360 a day, roughly $11,000 a month. A 30% exact-repeat rate takes that to about $7,700; a 5% rate takes it to $10,450. Substitute your own numbers. The engineering is the same in both cases, which is exactly why measuring the rate first is the whole decision.
Semantic caching, and why to be careful
Since exact repeats are often rare, the tempting extension is to serve a cached answer when a new question is similar enough to a previous one. This works, and it is a genuinely bigger saving. It also introduces a failure mode that ordinary caching does not have: two questions can be close in embedding space and have different correct answers. “What is the notice period for annual contracts?” and “What is the notice period for monthly contracts?” are extremely similar and not interchangeable.
If you do this, three safeguards are not optional. Set the similarity threshold conservatively, and tune it against real query pairs rather than by intuition. Never cache across users where permissions differ. And keep the negation and quantity cases in mind — small words that flip meaning are exactly what embeddings compress away, which is the same weakness described in when not to use RAG.
A semantic cache serving a subtly wrong answer is an invisible failure of the worst kind, per choosing an architecture by how it fails, and it will not show up in your monitoring. Weigh that against the saving with the error-cost arithmetic from the cost of being wrong.
Two rules that make answer caching safe
Key by identity as well as by question. If two users can see different documents, a shared answer cache is a permission leak. Include the authorisation context in the cache key, or restrict caching to content everyone may see. This is the failure that turns a cost optimisation into an incident.
Bound the lifetime by your freshness policy, not by convenience. A cached answer is a copy of a copy: it can outlive the document version it was drawn from. Whatever maximum staleness you committed to in choosing a freshness requirement you can afford applies to the cache too, and the deletion path must invalidate it. An answer cache that survives a document takedown is exactly the artefact you promised did not exist.
The order to implement them
- Reorder your prompt so the stable part comes first, and use prefix caching. Large return, no risk, roughly no work. There is no reason not to have done this already.
- Measure your exact-repeat rate. An afternoon. Decides everything that follows.
- If repeats are material, add an exact-match answer cache keyed by normalised question plus authorisation context, with a lifetime bounded by your freshness policy.
- Cache query embeddings and retrieval results if you are chasing latency rather than cost. Small and safe.
- Consider a semantic cache only if steps 1–3 left a large bill and repeats are near-misses rather than exact, and only with a conservatively tuned threshold, per-user keying, and monitoring of what it serves.
Most teams should stop after step 3, and a meaningful number should stop after step 1.
The recommendation
Do prefix caching immediately; it is free. The only requirement is prompt ordering, and it attacks the term that dominates a retrieval bill.
Measure the repeat rate before building an answer cache. It is the input that decides whether the work pays, and it varies enormously between internal and consumer-facing systems.
Treat the cache key as a security surface. Authorisation context in the key, always, or no caching across users at all.
Be sceptical of semantic caching. It converts a cost problem into a correctness risk, and the risk is invisible. Reach for it only when the arithmetic is compelling and you can monitor what it is serving.
The threshold that flips it: when your measured exact-repeat rate is above roughly a fifth of traffic, an answer cache is the single best return in your system and should be built before any other optimisation. Below that, spend the effort on prompt size and retrieval quality instead — see choosing a model size for the generation step, where the same money buys a permanent per-request reduction rather than a conditional one.