Scoping a Proof of Concept That Decides Something
A retrieval prototype over a handful of clean documents will work. That outcome carries almost no information, because the thing it demonstrated — that retrieval and generation function — was never in doubt. A pilot is worth running only if it can come back negative, and most are scoped so that it cannot.
Design the pilot around the specific uncertainty that is blocking your decision. That means naming the uncertainty first, which is the step usually skipped.
What the standard pilot proves, and what it hides
The standard pilot: pick 20 to 50 tidy documents, build the pipeline, demo a few impressive questions. It reliably succeeds and it leaves every real risk untested.
| Risk | Tested by the standard pilot? |
|---|---|
| Can we extract text from our actual documents? | No — the sample was chosen for being clean |
| Does retrieval find the right passage among thousands? | No — precision problems only appear at scale |
| Can we enforce permissions? | No — usually skipped entirely |
| Will users adopt it? | No — the demo audience is not the user |
| Can we keep it current? | No — the corpus was a snapshot |
| Can we tell whether it is working? | No — success was judged by vibes |
| Is the answer quality good enough for the stakes? | No — the questions were chosen by the builder |
Every row is a project-killer, and the pilot addressed none of them. Worse, a successful demo creates commitment: budget gets approved on the strength of a result that measured nothing, and the risks surface after there is a plan to defend.
Name the uncertainty, then design backwards
There are only a handful of genuine uncertainties in a retrieval project. Pick the one or two that would actually change your decision.
“Is our document set usable at all?” Common with scanned material, heavy tables, engineering drawings, or transcripts. The pilot is an extraction pilot: take the fifty worst documents, extract them, and have a domain expert score whether the extracted text preserves the meaning. No retrieval, no model, no UI. This is the cheapest pilot in this article and it kills more projects than any other — which is exactly why it should be first.
“Can retrieval find the right thing in our corpus?” The pilot is a retrieval pilot: index a realistic slice, collect 50 real questions with the correct source document noted by a human, and measure how often the right document appears in the top few results. No generation. If retrieval cannot find it, generation cannot fix it, and you have learned that in a fraction of the time.
“Will people use it?” The pilot is a product pilot, and it does not need to be good. A results page over a real corpus put in front of real users for two weeks tells you more about adoption than any amount of answer quality, and the query log it produces is the most valuable artefact any pilot generates.
“Is quality good enough for the stakes?” The pilot must be an evaluated pilot: questions written by someone other than the builder, answers scored blind by a domain expert against a rubric, failures categorised. This is the most expensive pilot and the only one that answers the question.
“Does the buy option suffice?” Then the pilot is an evaluation of the product, not a build — see the assistant you already pay for. Running a build pilot when the real question is buy-versus-build is an expensive way to avoid a procurement conversation.
The four rules that make a pilot informative
1. Use the real corpus, or a deliberately unfavourable slice of it. Not the clean subset. If anything, bias towards the ugly documents, because those determine feasibility. A pilot on curated material measures your curation.
2. Someone other than the builder writes the questions. This is the single highest-value rule. Builders unconsciously ask questions their system handles, and the effect is strong enough to invalidate an otherwise careful pilot. Take questions from a support queue, a search log, or a survey of intended users, and write them down before anyone runs them.
3. Define the pass condition in advance, numerically where possible. “The correct source document appears in the top five for at least 80% of the 50 questions.” Anything vaguer than that will be judged by whoever is most invested. Set the bar from the consequences, per the cost of being wrong.
4. Categorise every failure. The categories are the deliverable. Extraction failure, missing source, retrieval miss, right passage but wrong answer, and the question being unanswerable from the corpus at all. That distribution is your project plan; a single pass rate is not.
Time-box it and price it
Worked illustration, all inputs hypothetical. An extraction pilot is a couple of days. A retrieval pilot with a 50-question labelled set is one to two weeks including the labelling — which a domain expert does, not an engineer. A full evaluated pilot with blind scoring is three to four weeks. At a hypothetical loaded rate of $4,000 per engineer-week, plus domain-expert time, the evaluated pilot is roughly $15,000 to $20,000. Substitute your own rates.
Against the full-project estimate from pricing a RAG project before you have requirements — which for a modest system was in the low hundreds of thousands over a first year — a pilot at that price is cheap insurance if and only if it can return a negative. If it cannot, it is not insurance; it is the first four weeks of the project with a different name.
What a good pilot leaves behind
The pass or fail is the least valuable output. Three artefacts outlive the pilot:
The labelled question set. Fifty real questions with correct sources noted. This becomes your eval set, and it is the asset that makes every later component decision affordable — changing embedding models, chunking, or stores, per embedding model lock-in and what switching costs.
The failure distribution. Tells you where the work is. If most failures are extraction, your project is a document-processing project. If most are “not in the corpus,” it is a content project, and no amount of retrieval engineering will help.
A realistic cost sample. Real token counts per answer from real questions, which turns your cost model’s assumptions into measurements.
The recommendation
Write down the uncertainty the pilot must resolve, and the pass condition, before building anything. If you cannot state a result that would stop the project, do not run a pilot — you have already decided, and you should skip to scoping the build honestly.
Run the extraction pilot first when your documents are difficult. Days, no model involved, and it is the fastest available route to a well-founded no.
Never let the builder write the questions. Everything else on this list is a refinement; this one is the difference between a measurement and a demonstration.
The threshold that flips it: if the pilot’s failure distribution is dominated by anything other than retrieval quality — extraction, missing content, no corpus owner — then the retrieval project is not the project you need, and the honest next step is the one in when not to use RAG rather than a second, better-tuned pilot.