Do You Need Generation, or Just Better Search?
The R in RAG is often the whole value. If your users’ real problem is that they can’t find the right document, retrieval solves it and generation adds cost, latency, and a new category of wrongness on top. A ranked list of results with good snippets is a legitimate product, and it is dramatically cheaper to build, run, and defend.
The decision is not “chatbot or nothing.” It’s a choice between three products that share the same retrieval layer.
The three shapes
Search. Query in, ranked results out, each with a snippet showing why it matched. No model call at answer time. Cost per query is a search operation. Wrongness is bounded: the worst case is an unhelpful list, which users recognise instantly and correct by rephrasing.
Search with extraction. Ranked results, plus the specific passage that answers the question, highlighted in place. Still no generation — or a very small model doing span selection. The user reads the source’s own words.
Generated answer. Retrieved passages go to a model, which synthesises prose with citations. Cost per query includes input and output tokens. Wrongness is unbounded in a specific way: the answer can be fluent, confident, well-cited, and wrong, and users cannot tell by looking.
These are increasing in cost and in risk, and the increase is not gradual. The step from extraction to generation is where the money and the exposure both arrive.
What generation actually buys
Be precise about this, because it’s easy to pay for it and get nothing.
Synthesis across sources. When the answer requires combining three documents — a policy, an exception to it, and an effective date — a list makes the user do the combining. This is the strongest case for generation and it’s a real one.
Format transformation. The source is a dense table; the user wants a sentence. The source is a fifty-page contract; the user wants the notice period. Generation is genuinely good at this.
Handling the user’s phrasing. Users ask in their words, documents are written in the organisation’s words. Retrieval already bridges much of that gap; generation closes the rest by answering the question as asked rather than as indexed.
Conversation. Follow-ups that depend on what was just said. A search box has no memory and doesn’t want one.
And what it doesn’t buy: accuracy. Generation doesn’t make retrieval better. If the right passage wasn’t retrieved, generation produces a confident answer from the wrong material, which is strictly worse than a list that visibly lacks the answer.
The test
Ask what the user does with the answer.
| If the user’s next action is… | Build |
|---|---|
| Opening the document and reading around the answer | Search with snippets |
| Copying one fact into a form or a message | Search with extraction |
| Making a decision from several sources at once | Generated answer |
| Asking a follow-up that depends on the last one | Generated answer |
| Forwarding the source to someone else as evidence | Search — the document is the deliverable |
The forwarding row catches a lot of internal tools. In compliance, legal, finance, and support, the artefact people need is the authoritative document, not a paraphrase of it. A generated summary they then have to verify against the source has added a step to their job.
The cost difference, in shape
Search cost per query is a database operation — small, flat, and largely independent of how long the documents are. Generated-answer cost per query is:
(prompt tokens + output tokens) × your price per token
+ the same search operation underneathBoth terms scale with traffic, and the token term dominates. So the two products have the same cost at zero traffic and diverge linearly forever. At internal-tool volumes the difference may be immaterial; at consumer volumes it’s the entire budget.
Worked illustration with hypothetical inputs. Take 20,000 queries a day. Search: the marginal cost is close to your existing infrastructure. Generation on top: at a hypothetical $3 per million input tokens and $15 per million output, with 3,500 input and 400 output tokens per answer, that’s 70M input and 8M output tokens a day — $210 plus $120, so about $330/day, roughly $10,000/month. Substitute your own prices; the structure is what matters. Now ask whether the synthesis is worth ten thousand a month, because that is the actual question, and the honest answer for a document-finding use case is often no.
The latency difference is the same story: search returns in tens of milliseconds, a generated answer in seconds. See latency budgets for retrieval for where those seconds go.
The staged approach that almost always wins
Build search first. Not as a compromise — as the first stage of the same project.
- Ship retrieval with a results UI. You need a good retriever for either product, so this is not throwaway work. It’s the foundation, deployed.
- Instrument it. Which queries return nothing useful? Which ones get reformulated? Where do users click, and how far down? This is the cheapest requirements research you will ever do, and it comes from real users rather than a workshop.
- Add generation where the data says synthesis is needed. Often that’s a subset — one question type, one part of the corpus. Route those to a generated answer and leave the rest as search.
This ordering has three advantages. You have a working product in weeks. Your retrieval quality is measured before generation hides it. And when you do add generation, you know what it’s for, which means you can tell whether it’s working.
The failure mode of the opposite order — chatbot first — is that retrieval problems arrive disguised as model problems. Teams then spend months on prompts to fix a ranking bug they would have seen immediately on a results page.
The recommendation
If your users are trying to find a document, build search. Snippets and highlighting, not a chat box. It’s cheaper by an order of magnitude at volume, it’s faster, it’s explainable when it’s wrong, and it puts the authoritative source in front of the person who needs it.
If your users are trying to get an answer that no single document contains, build generation — and accept the cost and the verification burden as the price of synthesis.
If you don’t know which one you have, build search and find out. Two weeks of query logs will tell you what a quarter of speculation won’t.
The threshold that flips it: when a measurable share of queries — say, more than a fifth of real traffic — need material from two or more documents combined to be answerable, generation has earned its place. Below that, you are paying per query for a formatting convenience. And if what you actually need is a fact from a table, neither of these is the right build: see when not to use RAG.