Choosing a Model Size for the Generation Step

The generation model is usually the largest per-request cost in a retrieval system, and the choice of which one to use is often made by default — whichever was in the prototype. That is worth revisiting, because a well-grounded prompt asks the model to do a narrower job than a bare question does, and narrower jobs are where smaller models are most competitive.

The decision is not “which model is best.” It is “which is the cheapest model that does this job acceptably, and which questions need something else.”

What the model is actually being asked to do

Retrieval changes the task. Without retrieval, the model must recall facts and reason about them. With retrieval, the facts are in front of it, and the job decomposes into:

Reading comprehension over the supplied passages. Finding the relevant sentence in what you provided. Genuinely easy relative to recall, and smaller models handle it well.

Faithfulness — not adding anything. Staying inside the supplied material. This is a discipline rather than a capability, and it correlates with instruction-following more than with size.

Synthesis across passages. Combining several sources, reconciling a general rule with an exception, noticing a date makes one source obsolete. This is where capability starts to matter.

Judgement about sufficiency. Recognising that the passages do not contain the answer and saying so instead of producing something plausible. In practice the hardest of the four, and the one where stronger models tend to be more reliable.

Note the pattern: the two easy jobs are the bulk of the traffic, and the two hard ones are a minority. That asymmetry is the whole argument for routing rather than choosing.

The cost structure

Cost per answer = (retrieved tokens + instructions + question) × input price
                + answer tokens × output price

Two things follow. First, in a retrieval system the input term usually dominates, because the retrieved passages are large relative to the answer — which means model pricing per input token matters more than output pricing, the reverse of a chat product.

Second, prompt size is under your control in a way model choice is not. Retrieving fewer, better passages reduces cost on every request, permanently, with no capability trade-off — provided retrieval is good enough to make the smaller selection the right one. Improving retrieval is therefore a cost lever as well as a quality lever, and it is the one that compounds.

Worked illustration, all inputs hypothetical. Take 20,000 answers a day at 4,000 input and 400 output tokens. At a hypothetical premium price of $3 per million input and $15 per million output: 80M input and 8M output daily, so $240 plus $120 — about $360 a day, roughly $11,000 a month.

Now a hypothetical smaller model at $0.30 input and $1.20 output: $24 plus about $10, so $34 a day — about $1,000 a month. Substitute your own prices; the ratio between tiers is the durable part, and it is large.

An order of magnitude is enough to justify measuring whether the smaller model is adequate. It is not enough to justify assuming it.

How to decide, rather than guess

The decision needs evidence and the evidence is cheap to get, because you can run the same prompts through both models.

Take 50 real questions, written by someone other than the builder, with the same retrieved passages for each model. Holding retrieval constant is the whole point — otherwise you are measuring two things.
Score blind against a rubric that separates correct, incomplete, unfaithful (said something not in the passages), and should have refused.
Compare the failure categories, not the overall score. This is where the useful signal is.

The result usually has a characteristic shape: the smaller model is close on straightforward extraction and worse on the sufficiency judgement — it answers when it should decline. That is exactly the failure profile that matters for the cost of being wrong, per the cost of being wrong, and it is why a naive average score would have misled you.

Routing, which is usually the answer

Given a majority of easy questions and a minority of hard ones, sending everything to the expensive model overpays and sending everything to the cheap one under-delivers. Three routing patterns, in ascending order of cost and capability:

Route by question type. Cheap model for straightforward lookups, strong model for anything requiring synthesis across sources. Requires classification, which is itself a cheap model call, and gives you a place to enforce policy — such as always using the strong model for anything about money.

Route by retrieval confidence. When retrieval returns one clearly relevant passage, the job is extraction: use the cheap model. When results are scattered or weak, the job is judgement: use the strong one, or refuse. This has the appealing property of spending more where the system is least sure.

Escalate on self-assessment. The cheap model answers and flags low confidence; flagged cases go to the strong model. Costs two calls on the escalated subset and adds latency to exactly the requests that are already hardest — check the accounting in latency budgets for retrieval before choosing this one.

Pattern	Extra latency	Complexity	Good when
Single strong model	None	None	Low volume, high stakes
Route by question type	One classification call	Moderate	Question types are distinguishable
Route by retrieval confidence	None	Low	Retrieval scores are informative
Escalate on self-assessment	A second call, sometimes	Moderate	Hard cases are rare and expensive to get wrong

The confidence-based route is underrated: it costs almost nothing to implement, adds no latency, and reuses a signal you already have.

The interaction with retrieval quality

Worth stating plainly because it inverts the usual instinct: better retrieval makes a smaller model viable. If the right passage is reliably in the top few results, the generation job is extraction, and extraction is where cheap models are strongest. If retrieval is noisy, the model is doing selection as well as extraction, and selection is where capability shows.

So a team facing a large model bill has two options — pay for the bigger model, or improve retrieval and drop a tier. The second is a one-off engineering cost against a permanent per-request saving, and it improves answers rather than just relocating the cost. It is generally the better investment, and it is another reason the eval set from scoping a proof of concept that decides something pays for itself.

The recommendation

Start on a capable model, then measure whether you can drop a tier — do not start cheap and debug quality. Beginning with a strong model gives you a quality ceiling to compare against; beginning cheap leaves you unable to tell whether a problem is retrieval or generation.

Route by retrieval confidence as the default optimisation. It is nearly free, adds no latency, and spends your money where the system is least certain.

Treat retrieval improvement as a cost-reduction project. Fewer, better passages cut the dominant input term and can move you down a pricing tier at the same time.

Hedge everything here with “as of this writing.” Model tiers and prices move constantly; the structure — input tokens dominate, easy questions are the majority, sufficiency judgement is the hard part — moves far more slowly.

The threshold that flips it: when your monthly generation spend exceeds roughly a week of engineering time, it is worth a week of engineering on routing and prompt size. Below that, leave it alone and spend the week on retrieval quality instead. Compare the terms in what a RAG system actually costs to run before deciding which lever to pull.