Is RAG Still Worth Building?
Every few months something arrives that is announced as the end of retrieval. Longer context windows. Models that search the web themselves. Agents that plan their own lookups. Each time, a version of this question lands on a technical lead’s desk: are we about to spend two quarters building infrastructure that will be obsolete before it is paid off?
It is a fair question and it deserves better than tribal loyalty in either direction. The way to answer it is not to predict model capability. It is to separate the parts of a retrieval investment that are durable assets from the parts that are perishable code, and then check whether the durable share is large enough to justify starting.
The four things that do not go away
Your corpus is yours and the model has never seen it. No amount of general capability gives a model your contracts, your incident history, or last Tuesday’s policy change. Something has to put your material in front of it at answer time. That is retrieval, whatever it is called and however it is packaged.
Permissions are a per-request decision. Who may see what changes between users and changes over time. A capability that reads everything cannot enforce that; a system that decides what to fetch, per request, can. This one is structural rather than technical, which is why it has survived every generation of the argument.
Freshness has a clock. If your answer must reflect a document edited an hour ago, something must have noticed the edit. Bigger windows do not notice edits. See choosing a freshness requirement you can afford for what the clock costs.
Attribution is a product requirement, not a model feature. Users in regulated, medical, legal, and financial contexts need to be pointed at the source. A system that fetched a document can cite it. A system that absorbed a corpus months ago cannot, and asking it to try produces the plausible-looking reference that is worse than none.
Those four are why the category persists. Notice that none of them are about the model being weak. They are about ownership, authorisation, time, and provenance — and improvements in model quality do not touch any of them.
The three things that genuinely have eroded
Being honest about this is what makes the rest credible.
Naive retrieval over a small stable corpus. This used to be the default build and it is now often the wrong one, because the corpus fits in the prompt. The arithmetic is in RAG versus long context, and it moved against retrieval — genuinely, and permanently for corpora of that size.
A great deal of hand-written pipeline scaffolding. Query rewriting, multi-step planning, deciding whether a second lookup is needed — a lot of this was application code and is increasingly something a capable model does acceptably if you let it. If your plan’s differentiation is that scaffolding, your plan is perishable.
The retrieval stack as a set of independently procured products. Consolidation is real, and a component you chose because it was the best standalone tool in its category may end up bundled into something you already pay for. Which is an argument about build versus buy, not about whether to retrieve.
So the accurate statement is not “RAG is dead” and not “RAG is fine.” It is: the reasons to retrieve got sturdier and the amount of bespoke machinery required to do it got smaller. That is a good combination for anyone deciding now, and a bad one for anyone whose value was in the machinery.
Sorting your own plan into durable and perishable
Take the project as scoped and put every line item in one of two columns.
| Durable | Perishable |
|---|---|
| A clean, owned, deduplicated corpus | Chunking strategy code |
| Document metadata and permission mapping | Retriever and reranker choices |
| An eval set built from real user questions | Prompt scaffolding and orchestration |
| Knowing which questions users actually ask | The specific vector store |
| The ingestion path to your systems of record | Framework and SDK glue |
Everything in the left column keeps its value if the architecture changes completely. A curated corpus with owners and permissions is worth exactly as much to a preloaded-context system, or to an agent with a search tool, as it is to a vector index. An eval set of real questions is the asset that lets you switch architectures at all — without it you cannot tell whether the new thing is better.
Everything in the right column is replaceable, and should be built to be replaced: thin, boring, and behind an interface. If your estimate is mostly right-column work, that is not an argument against retrieving. It is an argument against your design.
The decision procedure
Ask three questions in order.
- Do any of the four durable reasons apply to us? Own corpus, per-user permissions, freshness clock, attribution requirement. If none apply — a small public stable corpus, one permission tier, no citation need — you probably do not need retrieval, and you should read when not to use RAG rather than this.
- What fraction of our estimate is left-column work? If it is most of it, build now: you are buying assets. If it is a minority, redesign until it is a majority. Corpus work first, machinery last.
- What would have to become true for us to stop needing this? Write the answer down. If it is “context windows get ten times larger and cheaper”, that is a real possibility and you should keep the corpus portable. If it is “permissions stop mattering” or “we stop needing to cite sources”, you have your answer and waiting is not a strategy.
What waiting actually costs
The case for waiting is that the ground is moving. The case against is that the durable work takes the longest and does not depend on which way it moves.
If you wait a year, you arrive with the same messy corpus, no metadata, no permission map, and no eval set — and whatever architecture is fashionable then will still need all four. The teams that adopt a new retrieval approach quickly are not the ones who guessed right about the architecture. They are the ones who had already done the left column.
The recommendation
Build the left column now, regardless. Corpus quality, ownership, metadata, permission mapping, and an eval set from real questions. None of it is architecture-specific, all of it is on the critical path of anything you do later, and it is the work that cannot be bought in a hurry.
Build the right column thin and expect to replace it. Assume the chunker, retriever, and orchestration you pick will be swapped within a couple of years. That assumption should shape the interfaces, not delay the project.
Do not wait for the architecture question to settle. It will not settle, and the waiting does not accumulate anything.
The threshold that flips it: if none of the four durable reasons apply to you, retrieval is genuinely optional and probably wasteful — take the simpler path. If two or more apply, the category is not going anywhere for you specifically, and the only remaining question is how much of your plan is perishable. If you build it and the ground still moves out from under it, retiring it cleanly is a manageable project when the corpus was the asset and the pipeline was just plumbing.