When to Retire a RAG System

Retrieval systems rarely get switched off. They get quietly ignored — traffic drifts down, the index goes stale, and the thing keeps running and keeps answering, wrongly, to whoever still finds it. That is the worst of the available outcomes: you pay the infrastructure and the residual attention, and you carry the risk of confidently wrong answers, in exchange for nothing.

Decommissioning is an architecture decision like any other, and it deserves the same arithmetic as the decision to build.

Five signals it is time

1. Usage has decayed to a handful of users. Retrieval systems have real fixed costs — the storage, the standing fractional staffing, the on-call. At low usage the cost per useful answer becomes absurd, and no amount of per-request efficiency helps, because the cost is not per request. Divide your monthly total cost by monthly answers; if that number is larger than what a person would charge to answer the question, the system is a luxury.

2. Nobody has owned the corpus for two quarters. The corpus owner left, moved teams, or was never really assigned, and no one has curated the index since. This is terminal rather than degraded, and the reason is in corpus drift and what it costs you: an unowned index does not hold steady, it decays, and its answers get worse in ways users cannot detect.

3. The questions moved to a system of record. The team built the API you did not have when you started. Now the authoritative answers are one call away, and your index is a stale replica of them. That is the trigger to move those question types to tool calls per RAG or tool calls — and if that is most of your traffic, the retrieval system has been superseded rather than merely improved upon.

4. The corpus shrank into the context window. Documentation got consolidated, or context windows grew, and the material that needed retrieval now fits in a prompt. The whole architecture was a response to a constraint that no longer binds, and continuing to pay for chunking, embedding, an index, and its maintenance is paying for a workaround. Run the arithmetic in RAG versus long context again — with today’s numbers, not the ones from when you built it.

5. It cannot be extended to what is now being asked of it. Permissions were not in the original design, or single-tenancy was assumed, or the ingestion is batch-only and immediate deletion is now required. Retrofitting any of those is a rebuild. Recognising a rebuild as a rebuild — rather than as a series of increments — is often what makes retirement the honest recommendation.

The arithmetic for shutting down

Annual cost of keeping it =
    infrastructure
  + permanent fractional staffing
  + the expected cost of wrong answers it still produces

Annual value =
    answers served × value per answer

The third term in the first line is the one people omit, and it is frequently the largest. A neglected retrieval system does not become inert; it becomes a source of confidently stale answers, and the sizing method for that is in the cost of being wrong.

Worked illustration, all inputs hypothetical. Suppose a system now serves 400 answers a month. Infrastructure of $300, plus 10% of an engineer at a hypothetical $16,000 loaded — $1,600 — is $1,900 a month, about $4.75 per answer. Add an error term: at a 5% materially-wrong rate on a stale index and a hypothetical $50 per uncaught error, that is another $1,000 a month, so nearly $7 an answer. Substitute your own figures.

At those numbers the question is not whether to optimise it. It is whether 400 answers a month justify $2,900, and whether the people still using it would be adequately served by something simpler.

Retire the architecture, keep the value

The mistake in the other direction is switching off a system that people depend on. Almost always there is a cheaper architecture that preserves most of the value, and the useful framing is demotion rather than deletion.

From	Demote to
Generated answers over a stale index	Search with snippets over the same corpus — visible failure, less maintenance
Retrieval over a small consolidated corpus	Documents in the prompt; no index to maintain
Retrieval over exported database records	Tool calls against the system of record
A custom system over suite documents	The assistant bundled with the suite, per the assistant you already pay for
A broad index nobody curates	A narrow index over the subset that has an owner

Each row removes a permanent staffing obligation while keeping users able to find things. The first row is particularly worth noting: turning generation off and leaving search on is a legitimate, cheap end state, not a failure, and it converts an invisible failure mode into a visible one at the moment you stop watching.

Doing it properly

Five steps, and the third is the one that gets skipped.

1. Announce it with a date, and tell the users you can identify. Retrieval systems accumulate quiet dependencies — someone’s weekly process, a support macro.
2. Redirect rather than 404. Point the URL at whatever replaced it, or at the underlying document source. Users who bookmarked it should land somewhere useful.
3. Delete the derived data, deliberately and verifiably. Chunks, vectors, cached answers, logged prompts containing document content. This is a real obligation, not tidiness: a decommissioned index is still a copy of your corpus, and cached answers are copies of copies — see caching and what it does to a RAG bill.
4. Keep the eval set. Those labelled questions with correct sources are the most durable artefact the project produced, and they are directly reusable by whatever comes next.
5. Write down why you shut it down. The next person to propose this system will make the same case that was made the first time, and the failure reason — usually corpus ownership, not technology — is what should shape their plan.

The recommendation

Give every retrieval system a review date at launch, and put the cost-per-answer calculation on the agenda. A system with no scheduled review will never be retired; it will only be abandoned, which is the same cost with added risk.

Shut it down when the corpus has no owner. This signal is worth more than the usage numbers, because an unowned index is not merely underused — it is actively producing wrong answers, and it will keep doing so until someone notices.

Demote before you delete. Search over the same corpus, documents in a prompt, or tool calls against a system of record will serve most of the remaining users at a fraction of the maintenance. Choosing the lower-maintenance architecture on the way down is the same decision as choosing it on the way up, per when not to use RAG.

The threshold that flips it: when the monthly cost of keeping the system — infrastructure plus staffing plus expected error cost — exceeds what it would cost to have a person answer the same volume of questions by hand, retire it. That is a crude comparison and it is the right one, because it measures the thing the system exists to do rather than the thing it is technically capable of.