RAG or a Knowledge Graph?
Somewhere in most retrieval projects, a senior engineer says the words “this really wants a knowledge graph.” They are sometimes right. The trouble is that a graph is not a component you add to a retrieval system — it is a second corpus, derived from the first, with its own build cost, its own drift, and its own maintenance owner.
So the question is not “would a graph help?” It nearly always would, a little. The question is whether the class of question you actually need to answer is one that flat retrieval cannot reach at all, because that is the only case where the extra pipeline pays.
What each one is actually good at
Flat retrieval — chunk, embed, search — answers questions whose answer is stated somewhere in one or two passages. If a document says the thing, retrieval can find the document.
Graph retrieval answers questions whose answer is not stated anywhere and must be assembled from relationships. Which suppliers are exposed to a component that a subsidiary of ours manufactures. Every policy that references the clause we are about to change. How many of the incidents last quarter traced back to the same upstream service.
Note the shape of those. They are multi-hop, or they are aggregate, or both. No single chunk contains the answer, so no amount of better chunking, hybrid search, or reranking gets you there. That is the real boundary, and it is a boundary of kind rather than of quality.
| Flat retrieval | Graph retrieval | |
|---|---|---|
| Best at | Answers stated in a passage | Answers assembled from relations |
| Multi-hop questions | Weak — needs luck or several passes | Native |
| “How many / which all” questions | Unreliable; sees only top-k | Native |
| Build cost | Chunk + embed | Extraction, resolution, schema design |
| Maintenance | Reindex changed documents | Re-extract, re-resolve, migrate schema |
| Fails by | Missing a passage | Missing or wrong relations, silently |
| Skills needed | Common | Scarcer: modelling, graph querying |
The three costs a graph adds
Extraction. Something has to turn prose into entities and relations. If that something is a language model, you are paying a generation-sized cost per document rather than an embedding-sized one — and embeddings are already the cheap part of the pipeline, as the arithmetic in what a RAG system actually costs to run shows. Extraction over a corpus is closer in cost profile to answering a question about every document than to indexing it.
Entity resolution. “Acme Corp”, “Acme Corporation”, and “ACME” have to become one node, or your graph quietly contains three companies and every aggregate query over it is wrong. This is the part teams underestimate, and it is not a one-off: it recurs on every ingest.
The schema. A graph needs a model of what entities and relations exist. Somebody owns that model, defends it against the fourth team that wants a slightly different edge type, and migrates it when it changes. This is the cost that never ends, and it is organisational rather than technical.
Costing it, roughly
All unit prices below are hypothetical placeholders. Substitute your own; only the ratio matters.
Take a 20,000-document corpus. Suppose embedding a document costs a hypothetical $0.0002 and extracting entities and relations from it with a model costs a hypothetical $0.02.
Flat index build: 20,000 × $0.0002 = $4
Graph extract: 20,000 × $0.02 = $400Four hundred dollars is not the problem — that is a rounding error against a project budget, and it is the number people quote when they want approval. The problem is the other two columns. Corpus churn of 5% a month means re-extraction of a thousand documents monthly, forever, plus resolution passes over the whole graph when a name normalisation rule changes. And somebody senior spends recurring time on the schema, which at the fractional-person rates in the team you need to run retrieval in production is the dominant term by an order of magnitude.
So: price the graph as a permanent fractional headcount plus a schema owner, not as an extraction bill. Teams that price it as an extraction bill approve it and then discover the real cost in month four.
The test
Take the twenty questions your system must answer — the real ones, from users or from the stakeholder who is funding this. For each, ask: is the answer stated in one or two passages of some document, or does it have to be assembled?
- Nineteen stated, one assembled. Build flat retrieval. Answer the one by hand, or with a report, or tell the user it is out of scope. Do not build an extraction pipeline for 5% of a question list.
- Half and half. Build flat retrieval first and route. The assembled questions are usually answerable from structured data you already have — a database, a ticket system, a CMDB — reachable by a tool call rather than a graph. That path is cheaper and is covered in RAG or tool calls.
- Most of them assembled. You may not have a retrieval problem at all. You may have a data-modelling problem wearing a chatbot costume, and the honest recommendation is to build the graph as a data asset with its own value, then put retrieval on top of it later.
That last case is the one worth naming clearly, because it changes who should own the project. A graph built to serve one assistant is a liability. A graph that several systems query, of which the assistant is one, is infrastructure. Same artefact, entirely different justification, and only the second one survives a budget review in eighteen months.
The middle path most teams should take
You can get a meaningful share of the graph benefit without a graph: put the relations you care about in chunk metadata. Document type, owner, effective date, the entity it concerns, the contract it belongs to. Then filter and group on those. It is a fraction of the work, it needs no extraction model, and it converts some “which all” questions into a filter rather than a traversal.
This is not as powerful as a graph and it does not pretend to be. It is the option that sits between “flat retrieval” and “a second data platform”, and it is missing from most versions of this debate.
The recommendation
Default to flat retrieval with rich metadata. It covers stated-answer questions, which are the overwhelming majority of what users ask an internal assistant, and it adds no extraction pipeline and no schema owner.
Reach for structured data via tool calls before you reach for a graph. If the assembled questions are about entities your systems of record already track, the relations exist already in a database. Extracting them again from prose to build a graph is duplicating an asset you own.
Build a graph only when assembled-answer questions are the point of the project, and fund it as infrastructure with a named schema owner. Not as a feature of an assistant.
The threshold that flips it: when more than roughly a third of your must-answer questions require traversing relations that exist nowhere but in prose, flat retrieval is not a weaker option — it is the wrong one, and the arithmetic shifts from cost to feasibility. Below that line, the graph is a capability you are buying at permanent-headcount prices to serve a minority of traffic, and the same money spent on corpus quality returns more. If you are not sure which side you are on, that uncertainty is itself the answer: scope a proof of concept that decides something and let the question list settle it.