Corpus Drift and What It Costs You

Every retrieval system is at its best on the day it launches and gets worse from there unless someone spends attention on it. Nothing in the code has to change for this to happen — the corpus changes, the questions change, and the gap between them widens. Corpus drift is the most reliably underbudgeted cost in a RAG project, and unlike infrastructure it does not show up on any invoice.

If you are deciding whether to build retrieval at all, this is the cost that determines whether the system is still worth having in eighteen months.

Four kinds of drift, in order of how fast they bite

Content drift. Documents get updated, superseded, or contradicted. The index now holds two answers to the same question and no signal about which is current. This starts on roughly day one and never stops.

Coverage drift. Users ask about things the corpus never covered, or covers only in a document nobody thought to include. Every new product, policy, or process widens the gap. Nothing in your monitoring flags it, because the system happily retrieves the nearest-but-wrong material and generates a confident answer from it.

Query drift. The way people ask changes as they learn what the system is good at, and again when they learn what it is bad at. A retriever tuned on launch-week queries is tuned for a distribution that no longer exists. This one is slower but it invalidates your assumptions rather than just your data.

Model drift. The generation model behind your answers is updated by its provider, or you upgrade it deliberately, and behaviour you had tuned around changes. As of this writing this is a normal feature of building on hosted models rather than an exception, and it means your system can change without anyone on your team touching it.

Note which of these are technical problems: none of them. All four are the consequence of a system’s environment moving while the system stands still.

What the maintenance work actually is

Give the abstract cost a shape by naming the tasks:

Task	Trigger	Rough cadence
Re-ingesting changed documents	Source edits	Continuous, automatable
Deciding which of several versions is authoritative	Source edits	Continuous, not automatable
Removing content that should no longer be answerable	Policy or access change	Event-driven, urgent when it happens
Adding newly relevant sources	New products, teams, processes	Monthly-ish
Re-checking quality against real queries	Anything above, plus model updates	Monthly-ish, or gated on changes
Re-embedding the corpus	Embedding model change or re-chunk	Rare, expensive, forced

The first row is engineering and it automates well. The second row is judgement and it does not. That asymmetry is the whole reason drift is expensive: the parts you can automate are the cheap parts.

Pricing drift before you commit

You cannot forecast this precisely, but you can bound it, and a bounded estimate is enough to make an architecture decision.

Monthly drift cost =
    corpus-owner hours × their loaded hourly cost
  + quality-review hours × engineering hourly cost
  + occasional re-embedding cost, amortised

Worked illustration with hypothetical inputs. Suppose a mid-sized internal corpus needs four hours a month of a domain expert’s judgement and six hours of an engineer’s quality review. At hypothetical loaded rates of $80 and $100 an hour — substitute your own — that is $920 a month, about $11,000 a year, in work that produces no new features and that nobody will thank anyone for. Add a forced re-embed once a year and it is higher.

Put that number next to the infrastructure estimate in what a RAG system actually costs to run. For most internal systems, drift maintenance is the larger figure. Any comparison of architectures that omits it is comparing the wrong things.

How the alternatives drift

This is the part that makes drift a selection criterion rather than a tax you pay regardless.

Documents in the prompt drift only in content — you update the file and the next request uses it. There is no index to resynchronise, no stale copy to hunt, no re-embedding. This is a genuine and underrated advantage of the stuff-the-context approach for small corpora, and it belongs in the arithmetic in RAG versus long context.

A fine-tuned model drifts worst of all: correcting a fact means another training run, and until then the wrong answer is baked in. That asymmetry is most of the argument in RAG versus fine-tuning as an architecture decision.

Tool calls against a live system barely drift, because the database is the source of truth and it is already maintained by whoever owns it. You inherit someone else’s maintenance instead of creating your own — the strongest structural argument for tool calling wherever it fits.

Retrieval sits in the middle: much better than fine-tuning, worse than tool calls, worse than a small prompt corpus. Its drift is proportional to how many documents you indexed and how fast they change.

Designing for less drift

Three choices at design time reduce the permanent bill, and all three cost you something at launch.

Index less. Coverage feels free and is not. Every document added is a document that can go stale, contradict another, or get retrieved instead of the right one. A deliberately narrow corpus, expanded on evidence of real demand, has lower drift and usually higher answer quality.

Prefer sources that already have owners. A wiki space with a named maintainer drifts slowly. A shared drive folder drifts immediately. Choosing the maintained source over the complete one is the cheapest quality decision available.

Keep your ingestion reproducible. If you can rebuild the entire index from source documents with one command, re-embedding is an afternoon. If chunking rules live in a notebook or a vendor console, it is a project — which is why exit cost and drift cost turn out to be the same question, as in build or buy your retrieval stack.

The recommendation

Treat drift maintenance as a permanent line item and put a number on it before you approve the build. If the sponsor will not fund ten hours a month of ongoing attention, do not build retrieval — build something whose failure is visible, or narrow the scope until the maintenance fits the funding.

Choose the architecture whose drift profile matches your corpus. Fast-changing documents with a real owner: retrieval earns it. Facts that live in a database: tool calls, and inherit their maintenance. Small stable corpus: the prompt, and skip the whole category.

The threshold that flips it: when the corpus is large enough that no single person can tell you which document is authoritative, drift has already won, and adding retrieval will make wrong answers more fluent rather than less frequent. Narrow the scope until one person can answer that question, then index that.