Cache-Augmented Generation, and Whether You Qualify

Cache-augmented generation — CAG, in the acronym everyone reached for once the papers appeared — is the idea that if your entire corpus fits in a context window, you can process it once, keep the model’s internal state for it, and then answer every subsequent question against that preprocessed state. No retriever, no index, no chunk boundaries, no top-k. The model has already read everything.

When you qualify, it is the cheapest good architecture available, and it removes an entire failure class. The reason this is not the end of the retrieval industry is that qualifying is narrow, and the four preconditions are easy to assume you meet.

This is a different question from the general RAG versus long context cost arithmetic, which asks what you pay per request at a given corpus size. CAG asks something more specific and more binary: can you amortise reading the corpus across many requests, and does anything about your situation break the amortisation?

Precondition 1: the corpus fits, with headroom

Not “fits.” Fits with room for the conversation, the instructions, and the answer, and still fits after the corpus grows for eighteen months.

The failure mode is not a graceful degradation. On the day the corpus exceeds the window, CAG stops being an architecture and you build the retrieval pipeline you deferred, under time pressure, with a live product on top of it. Growth rate matters more than current size here: a corpus at 40% of the window growing 10% a month is not a CAG candidate, and a corpus at 70% that has been flat for three years is.

Precondition 2: the corpus is stable relative to the cache lifetime

Any cached prefix has a lifetime. Every edit to the corpus invalidates the cache and you pay to build it again.

That gives you the actual operating question: how many requests do you serve per corpus change? Call it Q. If Q is in the thousands, the write is free in effect. If Q is single digits — a corpus that changes several times an hour, an internal wiki with fifty editors — you are paying full price to read the corpus over and over, which is the worst version of both architectures at once.

Precondition 3: everyone may read everything

This is the precondition that disqualifies the most projects, and it is rarely in the initial design conversation.

A preloaded corpus is a single shared state. If two users are entitled to see different documents, you cannot serve them both from one cache — you would need one cache per permission set, and the number of distinct permission sets in a real organisation is closer to the number of people than to one. Filtering after the fact does not help: the model has already read the restricted document and can paraphrase it into an answer without quoting it.

Retrieval handles this natively, because permissions become a filter on what gets retrieved. CAG has no equivalent hook. If your corpus is public-facing documentation, a product manual, a body of law, or a policy set everyone in the company may read, you pass. If entitlements vary, you fail, and the constraint is the same one that governs data residency as an architecture constraint: it removes options rather than changing prices.

Precondition 4: you control, or can tolerate, the cache mechanics

Whether long-lived prefix caching is available to you, at what discount, with what expiry, and whether the cache write itself is billed, all depend on your provider and all move. As of this writing, treat none of it as stable and verify against your own contract.

Two consequences worth planning for regardless of vendor. Cold starts are real: the first request after an expiry pays the full read, so your latency distribution has a tail your average does not show. And you cannot pin a cache indefinitely unless you host the model yourself, at which point you have taken on the hosting problem in exchange for the retrieval problem.

The amortisation arithmetic

All unit prices below are hypothetical placeholders — substitute your own.

Suppose a 180,000-token corpus, a hypothetical $3.00 per million input tokens at full rate, and a hypothetical cached-read rate one tenth of that.

Full read (cache write):  180,000 × $3.00 / 1e6      = $0.54
Cached read per request:  180,000 × $0.30 / 1e6      = $0.054
Cost of N requests in one cache lifetime: $0.54 + N × $0.054

Per-request cost falls asymptotically toward the cached-read rate as N grows, and the interesting quantity is not the average — it is what happens when N is small. At N = 1 you paid $0.594 for one answer. At N = 100 you paid about $0.059 each. The whole architecture is a bet on N, and N is set by your edit frequency, not by your traffic.

Now compare against sending a few thousand retrieved tokens per request at the full rate — pennies of a cent — and the shape becomes clear: even a well-amortised CAG request carries the whole corpus on its back at a discount, where a retrieval request carries almost nothing at full price.

	CAG	Retrieval
Per-request input	Whole corpus, discounted	A few thousand tokens, full rate
Sensitive to corpus size	Strongly	Barely
Sensitive to edit frequency	Strongly	Barely
Per-user permissions	Not supported	Native
Infrastructure	None	Index, pipeline, ops
Retrieval-miss failures	None	Yes

The hybrid nobody names

The useful version for most teams is not either/or. Split the corpus by volatility: the stable core goes in the cached prefix, the long tail goes behind a retriever. Product fundamentals, terminology, the org’s standing policies, the schema — cache those. Last week’s tickets, the changing price list, the per-customer documents — retrieve those.

You get the no-miss quality of a preloaded core on the material every question touches, and you keep the permission hook and the size independence where you need them. The cost is that you now run both, which means you have not avoided the pipeline. That is the honest trade, and it is worth it more often than the pure form of either.

The recommendation

Test the four preconditions before you model any cost. Fit with headroom, stability relative to cache lifetime, uniform read entitlement, and tolerable cache mechanics. One failure disqualifies the pure architecture — they are not weighted criteria.

If you pass all four, do it, and do not build a retriever. A small, stable, wholly-public corpus served from a preloaded context is a better product than a retrieval pipeline over the same material, because it cannot fail to find something. Build the pipeline the month growth projections put you within reach of the window.

If you fail on permissions, stop considering it entirely. There is no configuration that fixes this and the workarounds leak.

If you fail only on size or volatility, build the hybrid — cache the stable core, retrieve the tail — rather than abandoning the idea. The prefix ordering that makes it work is free, and it is step one of caching and what it does to a RAG bill.

The threshold that flips it: when requests per corpus change drops below roughly ten, the amortisation stops working and CAG becomes the most expensive way to answer a question you have. Above a few hundred, on a corpus comfortably inside the window that everyone may read, it is the cheapest — and the pipeline you did not build is the maintenance you do not owe.