Build or Buy Your Retrieval Stack
Buying a retrieval platform removes plumbing. It does not remove the work that actually makes a RAG system good, and it does not remove the obligation to own the corpus. That is the whole decision in two sentences, and most build-or-buy debates go wrong because both sides argue about the plumbing.
Here is a way to run the decision that ends in an answer rather than a bake-off.
Separate the stack into layers you can buy independently
“Build or buy” is a false binary. A retrieval system has at least six layers, and you can make a different call on each:
| Layer | What buying gives you | What it can’t give you |
|---|---|---|
| Document extraction | Parsers for messy formats you’d otherwise write | Judgement about what to keep and discard |
| Chunking | Reasonable defaults | Chunk boundaries suited to your documents |
| Embedding | A hosted model and an API | Freedom from re-embedding when you switch |
| Vector storage & search | Operations, replication, backups | Your access-control model |
| Orchestration | Retrieval-then-generate glue | The prompt that fits your task |
| Evaluation | A metrics dashboard | A labelled set drawn from your real queries |
Nearly everyone buys the embedding model and nearly everyone builds the prompt. The genuinely contested layers are storage and orchestration. Decide those two on their merits and stop treating the rest as one purchase.
The four questions
1. Is your differentiation in the retrieval or in the product around it? If the answer your system produces is the product, you will end up tuning retrieval deeply, and a platform that hides the ranking behaviour becomes an obstacle within months. If retrieval is a feature inside a larger product — a help widget, an internal assistant — the platform’s defaults are probably better than what you’d build in the same time.
2. What does your access-control model look like? This is the question that most often forces a build. If every user sees every document, buy. If documents carry per-user or per-group permissions that change, you need filtering that reflects your authorisation system, and that integration is rarely a configuration setting. Ask specifically how a vendor handles a permission that was revoked five minutes ago.
3. How weird is your data? Standard prose in standard formats is a solved problem and you should not spend engineering time on it. Scanned forms, engineering drawings, transcripts with speaker turns, tables that carry the actual meaning — these break generic pipelines, and the extraction layer becomes your problem regardless of what you buy.
4. What happens when you want to leave? Not a theoretical concern. Vector data is portable in principle and awkward in practice: you can export vectors, but they’re only meaningful with the model that produced them, and vendors’ chunking and metadata conventions differ. The realistic exit cost is re-running your ingestion pipeline against a new store. If you kept your source documents and your chunking logic, that’s a few weeks. If your chunking only exists inside the vendor’s config, it’s a rebuild.
The cost comparison, honestly
The comparison people run is a platform’s monthly fee against zero, because self-hosting feels free. It isn’t, and the fee is not the interesting number either.
Set it up as three columns, with your own figures substituted for anything hypothetical:
Buy = subscription + per-request charges + integration engineering
Build = infrastructure + engineering to first release
+ ongoing operations (fraction of an engineer, per month, forever)The term that decides it is almost always the last one. A self-hosted vector store is not a lot of work per month — until it is, at 3 a.m., on the week two people are on holiday. Price it as a standing fraction of a person’s time rather than as an occasional cost, because an on-call rotation is a standing commitment even when the pager is quiet.
Worked illustration, all inputs hypothetical. Suppose a platform quotes you $1,500/month, and self-hosting the equivalent costs $400/month in infrastructure plus 10% of one engineer’s time. At a fully loaded engineering cost of, say, $16,000/month — substitute your real figure — that 10% is $1,600. Self-hosting is now the more expensive option, and you also carry the risk. Change the engineering figure or the utilisation and the answer flips; that’s the point. Run it with your numbers before the meeting, not during it.
The asymmetry worth noticing: buying is expensive in proportion to your usage, building is expensive in proportion to your headcount. Small team with high volume leans build. Large team with modest volume leans buy, which is the opposite of what most people’s instinct says.
What buying never removes
Three obligations survive every purchase decision, and they are the ones that determine whether the system is any good in a year.
Someone owns the corpus. A vendor cannot tell you that the returns policy changed, or which of three superseded documents is authoritative. This is the failure that kills retrieval projects, and no product addresses it. If you can’t name the owner, see when not to use RAG.
Someone owns quality measurement. A dashboard is not an eval set. Labelled examples of real queries with correct answers are yours to produce regardless of who hosts the index.
Someone owns the answer when it’s wrong. Users escalate to you, not to your vendor. The debugging path has to be walkable by your team, which means you need visibility into what was retrieved and why — a hard requirement to put in the procurement conversation.
The recommendation
Buy the storage and search layer unless you have a specific reason not to. Vector search operations are undifferentiated work for the overwhelming majority of teams, and the two legitimate reasons to self-host — a hard data-residency requirement, or volume large enough that per-request pricing dominates your bill — are both easy to recognise. If neither applies to you, running your own index is a hobby.
Build the orchestration. The retrieve-rerank-assemble-prompt path is where your product’s quality lives, it’s a few hundred lines, and owning it keeps every future tuning decision available to you. Frameworks are fine as a reference; a framework you can’t step through is not.
Buy nothing until you have run the arithmetic in what a RAG system actually costs to run, because the most common outcome of pricing the whole thing is discovering that the volume doesn’t justify either option yet.
The threshold that flips this: when integration work against the platform starts exceeding what the equivalent self-hosted work would cost — usually driven by permissions or by a ranking behaviour you can’t reach — stop paying twice and move that layer in-house. That’s a real signal, and it typically arrives well before the invoice becomes the problem.