Agentic Retrieval or a Fixed Pipeline?
A fixed retrieval pipeline does the same thing on every request: embed the question, search once, put the results in a prompt, generate. Its cost per question is a constant you can multiply by traffic and put in a spreadsheet.
An agentic retrieval loop lets the model decide. It reads the question, issues a search, looks at what came back, decides whether that was enough, searches again with a better query if not, and stops when it thinks it has the material. Its cost per question is a distribution, and so is its latency.
That is the whole decision, and it is a decision about variance rather than about capability. Almost everyone agrees the loop produces better answers on hard questions. Far fewer teams have priced what a distribution does to a budget and a capacity plan.
What the loop actually buys
Three things, and only three.
Multi-hop questions. Where the second search depends on what the first one returned. “Which of our suppliers is affected by the recall we announced in March” needs the recall document before you can search for suppliers. One pass cannot do this, no matter how good the retriever.
Recovery from a bad first query. When the user’s phrasing does not match the corpus vocabulary, a fixed pipeline returns nothing useful and generates from nothing useful. A loop notices the results are thin and tries different words. This is the most common real benefit and it is genuinely valuable.
Knowing when to stop. A loop can conclude that the corpus does not contain the answer and say so, because it has evidence from several attempts rather than one. A fixed pipeline sees one weak result set and cannot tell “not in the corpus” from “asked badly”.
What it does not buy: better answers to questions a single well-formed search already answers. For those — the majority of traffic on most internal assistants — the loop spends more money and more seconds to reach the same place.
The cost shape
Every unit price below is a hypothetical placeholder. Substitute your own; the shape is the point.
Suppose a fixed pipeline costs a hypothetical $0.012 per answer all-in. In a loop, each iteration adds a planning call and a read of the returned passages — say a hypothetical $0.009 per iteration — on top of a final answer generation.
Fixed: $0.012 per question, always
Agentic: $0.012 + (iterations × $0.009)Now the part that matters. Iterations are not a number, they are a distribution, and the mean lies about the bill because the tail is where the cost lives:
| Iterations | Share of traffic | Cost per question |
|---|---|---|
| 1 | 55% | $0.021 |
| 2 | 25% | $0.030 |
| 3 | 12% | $0.039 |
| 4–6 | 6% | $0.048–$0.066 |
| Hit the cap | 2% | cap cost |
Weighted out, that hypothetical set averages a little over double the fixed pipeline. Two facts follow, and both are more useful than the average.
First, the questions in the tail are not random — they are the hard ones, which correlate with your most important users and your most sensitive queries. You cannot cheaply sample them away.
Second, the distribution shifts when your corpus changes. A degraded index makes every first search worse, more searches follow, and the bill rises without any traffic change. An agentic system’s cost is a quality signal, which is genuinely useful, and it is also a budget that moves on its own.
Latency is the harder constraint
The bill you can absorb; the clock the user cannot. Each iteration is a serial round trip — a model call, a search, a read — so a three-iteration answer is roughly three times the retrieval-and-planning latency of a one-pass answer, before generation. Against the budgets in latency budgets for retrieval, that is the difference between a fast answer and an abandoned session.
There are only two honest ways to live with this. Stream the work so the user sees progress and the wait becomes legible rather than dead air. Or cap the loop hard and accept a worse answer at the boundary. Systems that do neither test fine at the median and fail at the tail, which is the failure profile that survives launch and then erodes trust quietly — the pattern in choosing an architecture by how it fails.
Route, do not choose
Framed as either/or, this decision is usually made wrongly, because the answer differs by question. The design that holds up is a fixed pipeline as the default path with an escalation to the loop, and the escalation triggered by evidence rather than by a classifier’s guess about difficulty.
The cheapest trigger is the retrieval result itself: run one search, and if the top results are weak — low scores, few of them, or none passing a relevance bar — escalate. You have then spent one search on the easy majority and reserved the loop for the cases where the first pass demonstrably failed. This is the same routing discipline that decides RAG or tool calls, applied to depth instead of to source.
Two controls belong in the design from the start, not added after the first surprising invoice:
- A per-question iteration cap, so no single request can cost an unbounded amount. Pick a number, log every request that hits it, and treat that log as your queue of questions the architecture cannot answer.
- A per-user or per-day spend ceiling. An agentic system with no ceiling has no maximum bill, and the first person to discover that is usually a script.
The recommendation
Default to a fixed pipeline and escalate on weak retrieval. It gives you a constant cost for the traffic that does not need more, and the loop’s benefit where it is demonstrable. This is the right answer for most internal assistants and most support products.
Go loop-first only when multi-hop questions are the product. Research tools, investigation workflows, anything where the user expects to wait because they are asking something genuinely hard. There the variance is not a defect, it is the shape of the work — and users will tolerate seconds if you show them progress.
Budget from the tail, never the mean. Capacity-plan on the p95 iteration count and price the monthly bill at a pessimistic distribution. An average-based estimate on a variable-cost architecture is not conservative, it is wrong in the direction that gets noticed by finance.
Cap iterations and cap spend on day one. Both are a few lines of code before launch and an incident afterwards.
The threshold that flips it: when more than roughly a third of your questions need a second search whose query depends on the first result, the fixed pipeline is not merely cheaper — it is unable to answer your product’s central question, and you should design loop-first with hard caps. Below that, escalate rather than commit, and keep the constant-cost path for the majority. Whichever way you go, add the iteration multiplier to the run-cost model in what a RAG system actually costs to run before you quote a monthly figure to anyone.