The Cost of Being Wrong
Every architecture discussion about retrieval eventually reduces to how much wrongness you can tolerate, and almost nobody quantifies it. That is unfortunate, because a rough figure for the cost of one bad answer determines the design more decisively than corpus size or query volume — it tells you whether to generate answers at all, whether a human belongs in the loop, and how much measurement you have to fund.
You do not need precision. You need an order of magnitude, and you can get one in a meeting.
Sizing it in four questions
1. Who bears the cost of a wrong answer? The user, your organisation, or a third party. If the user bears it and can detect it — an engineer misled by a stale doc for ten minutes — the cost is small and self-limiting. If your organisation bears it, it is a commitment you did not intend to make. If a third party bears it, you are in a different risk category and probably need review before publication.
2. Can the person receiving the answer tell it is wrong? This is the multiplier. A domain expert catches most errors and the cost approaches zero. A new customer cannot, so the error propagates into a decision. The same system has wildly different error costs for these two audiences, which is why “who is this for” is an architecture question.
3. Is the error recoverable, and how expensive is recovery? A wrong meeting-room booking is a minor annoyance. A wrong statement about a refund window becomes a commitment you honour or a complaint you handle. A wrong statement about eligibility or safety may be unrecoverable in the sense that matters.
4. How often does the class of wrong answer occur? You cannot know before building, but you can bound it: retrieval systems produce confidently wrong answers at a rate that is small but not negligible, and it will never be zero. Design as though the rate is a few percent of answers rather than a fraction of a percent, and check it after launch. Any plan that only works at zero is not a plan.
Turning that into a number you can use
Expected monthly error cost =
answers per month
× fraction that are materially wrong
× fraction not caught by the recipient
× cost per uncaught wrong answerWorked illustration, all inputs hypothetical. Take 30,000 answers a month, a materially-wrong rate of 3%, a recipient catch rate of 70% for a knowledgeable internal audience, and a hypothetical cost of $50 per uncaught error in wasted time and rework. That is 30,000 × 0.03 × 0.30 × $50 = $13,500 a month.
Now change one input — an external audience that catches only 20%: 30,000 × 0.03 × 0.80 × $50 = $36,000 a month. And if the cost per error is a hypothetical $500 because it involves a commitment about money, that same external case is $360,000 a month, which is not a system anyone should ship without a review step.
Every number there is invented; put yours in. The point is the structure and the sensitivity: the catch rate and the cost per error swing the result far more than the error rate does. So the highest-leverage design moves are the ones that raise the catch rate or lower the cost per error — not the ones that chase a marginally better retriever.
What each design move buys you
Read this as a menu priced against the number you just calculated.
Prominent source citations. Raises the catch rate, cheaply, because a user who can see and open the source can verify. The single best return in this list and it is mostly a UI decision.
Showing the source passage rather than a paraphrase. Raises the catch rate further, since the user reads the document’s own words. Reduces the appeal of the product slightly and the risk substantially — the trade explored in do you need generation, or just better search?.
Refusal on low retrieval confidence. Converts some wrong answers into no answers. Costs you coverage, and users complain about refusals more than they complain about errors, which is a preference you should be willing to override.
Scope restriction — declining whole categories. Anything about money, legal commitments, safety, or health goes to a human. Removes the expensive tail entirely, and it is the highest-value governance decision available. Write the categories down before launch.
Human review before the answer is sent. Drives the cost near zero and destroys the economics of automation. Sensible for low-volume, high-stakes flows; nonsensical at consumer volume.
A measurement function. Does not reduce error cost directly. It tells you what your rate and catch rate actually are, which converts every row above from speculation into a decision. If your calculated error cost is large, this is not optional — it is the cost of admission, and it is the staffing described in the team you need to run retrieval in production.
The decision table
| Error cost per uncaught wrong answer | Design |
|---|---|
| Trivial, recipient is expert | Generate freely, cite sources, measure occasionally |
| Moderate, recipient is expert | Generate, cite prominently, measure monthly |
| Moderate, recipient cannot verify | Show source passages; refuse on low confidence |
| High, any recipient | Restrict scope; route the expensive categories to humans |
| High and unrecoverable | Do not generate an answer — retrieve and let a person decide |
That last row is a legitimate outcome and worth saying explicitly: for some use cases the right product is a very good search tool for a trained person, not an answering machine for an untrained one.
Where this fits in the build decision
Two consequences for the overall economics.
It changes the comparison, not just the safeguards. An architecture with legible failure — keyword search, retrieval with a results page, tool calls against a system of record — has a much lower error cost for the same accuracy, because the catch rate is structurally higher. Once error cost is a term in your comparison, those options score better than they do on accuracy alone, which is the argument in choosing an architecture by how it fails.
It can exceed the infrastructure cost by an order of magnitude. In the hypothetical above, the error cost dwarfed any plausible token bill. If that holds with your numbers, then your optimisation target is the catch rate, not the cost per query — and a project plan focused on efficiency is optimising the small term. Compare against what a RAG system actually costs to run and see which is larger for you.
The recommendation
Do the four-question exercise before choosing an architecture, and write the number in the design document. Even a bad estimate reframes the project, because it puts the risk in the same units as the budget.
Spend on the catch rate first. Citations, visible source passages, and honest confidence signals are cheap, and they attack the term that dominates the arithmetic. Retriever tuning is a smaller lever than the interface is.
Enumerate the categories you will refuse, in writing, before launch. This is the move that removes the expensive tail, and it requires a product decision rather than an engineering one — so it has to be made by someone with the authority to make it.
The threshold that flips it: when the estimated monthly error cost exceeds the monthly cost of putting a person in the loop for the risky subset, put the person in the loop for that subset. That is an arithmetic answer to what is usually treated as a philosophical question, and it is generally cheaper than either extreme.