Scaling Thresholds for a Retrieval System
Retrieval systems do not degrade smoothly as they grow. They work fine, and then one variable crosses a line and a design decision that was correct becomes wrong. Knowing where those lines are is what lets you choose an architecture you will not have to replace, and — just as usefully — decide to ignore a scaling concern that is years away.
There are four variables that move: corpus size, query volume, update rate, and the number of distinct audiences. Each has its own threshold and its own redesign.
Corpus size: the two lines that matter
The first line is the context window. Below it, retrieval is optional; above it, retrieval or something like it is mandatory. That is the cleanest architectural threshold in the whole subject, and it is the one most projects are nowhere near.
The second line is much further out and softer: the point where flat similarity search over everything stops returning the right thing often enough. As a corpus grows, the number of passages that are plausibly similar to any query grows with it, and precision falls even though nothing about your retriever changed. The symptom is a system that got worse after you added a large source, with no code change to blame.
The redesign at the second line is not a bigger index. It is partitioning: filtering by metadata before the similarity search runs, so the search happens over the relevant slice rather than the whole corpus. Deciding what those slices are is a content-modelling exercise, and it is far easier to do early, while you still know what your documents are.
| Corpus regime | Right architecture |
|---|---|
| Fits in the context window comfortably | Put it in the prompt; see when not to use RAG |
| Too large for the prompt, one coherent subject | Flat retrieval, single index |
| Large and heterogeneous | Retrieval with metadata filtering, and route queries to slices |
Query volume: where the cost curve bends
Volume does not usually break retrieval technically. It breaks the budget, and it breaks whichever component you sized for a demo.
The important thing about volume is that it changes which architecture is cheapest, and the change is a crossover rather than a gradient. Prompt-stuffing costs scale with corpus size times request count; retrieval costs scale mostly with request count alone, plus a fixed storage term. At low volume the fixed term dominates and stuffing wins. At high volume the token term dominates and retrieval wins by a lot. The arithmetic is in RAG versus long context, and the practical advice is to know roughly where your crossover sits so that an increase in traffic is a planned migration rather than a surprise invoice.
The second volume threshold is concurrency at the generation step. Retrieval scales cheaply and horizontally; generation is where your rate limits and your queueing live. When traffic grows, the retrieval half of the system usually keeps up and the model half does not — so the redesign is caching, smaller models for the easy questions, or admission control, not a bigger vector store.
Update rate: when batch reindexing stops being enough
A nightly rebuild is the simplest possible ingestion design and it is correct for a surprising range of systems. It stops being correct at two distinct points.
The first is when the business consequence of staleness exceeds the rebuild interval — a price, an entitlement, an on-call rota. Once a wrong answer between rebuilds is a real problem, you need incremental updates, which is a substantially more complex pipeline with a new failure mode: partial state. That is a threshold worth resisting, and often the better answer is to move the volatile facts out of retrieval entirely and fetch them live with a tool call.
The second is when deletion becomes a requirement rather than a nicety. An access revocation or a takedown that must take effect immediately is not satisfiable by a nightly job, and retrofitting it is unpleasant. If you can see that requirement coming, design for it now.
Choosing this deliberately rather than by default is the subject of choosing a freshness requirement you can afford.
Audiences: the threshold that is really about permissions
One index serving one group of users who can all see everything is an easy system. The threshold arrives the first time two users must get different results from the same query.
Once that is true, every retrieval call needs authorisation context, your quality measurement needs to account for who was asking, and your cache keys are no longer just the query. This is a genuine architectural break rather than a feature, and it is the most common reason a working internal pilot cannot be extended to a second department. Whether you handle it with one filtered index or several is the trade-off in one index or many?.
A worked look at how far away these are
All inputs hypothetical; substitute your own. Take a corpus of 5,000 documents averaging 1,200 tokens, so roughly 6 million tokens. That is far past any prompt-stuffing approach, so retrieval is mandatory on corpus size alone. Now take 2,000 queries a day. At a hypothetical $3 per million input tokens with 4,000 tokens of prompt per answer, the inference input cost is around $24 a day — a few hundred dollars a month, plus output. Storage for a corpus that size is a rounding error on either self-hosted or managed pricing.
The point of the illustration is the ratio, not the numbers: at these volumes you are nowhere near a cost threshold, so any redesign should be driven by precision, freshness, or permissions instead. Teams routinely optimise the cheap axis. Run your own version of that estimate before choosing what to work on.
The decision table
| Variable | Threshold | What changes |
|---|---|---|
| Corpus size | Exceeds the context window | Retrieval becomes mandatory |
| Corpus size | Heterogeneous enough that precision falls | Metadata filtering and routing |
| Query volume | Token cost exceeds the fixed infrastructure cost | Retrieval beats stuffing; then caching |
| Query volume | Generation concurrency limits bite | Caching, model tiering, admission control |
| Update rate | Staleness has a business consequence | Incremental ingestion instead of batch |
| Update rate | Deletion must be immediate | Ingestion pipeline redesign — design in advance |
| Audiences | Two users need different results | Permission-aware retrieval; possibly split indexes |
The recommendation
Design for the threshold you are within one order of magnitude of, and explicitly ignore the rest. Write down your current numbers for all four variables and your projections. Anything more than 10× away is not an input to today’s decision, and pre-building for it is the most expensive form of optimism in this field.
Two of these are worth pre-empting even when they are far away, because retrofitting them is disproportionately painful: immediate deletion, and per-user permissions. If either is plausibly in your future, spend the design effort now.
The threshold that flips everything else: permissions. Corpus growth and volume growth are gradual and mostly buyable. The first time two users must see different results, you have a different system — so decide before you build whether that day is coming, and if it is, treat the single-audience version as a prototype rather than a first release. Before any of this, confirm the whole thing is worth building: what a RAG system actually costs to run.