RAG or Tool Calls: Which One Does Your Question Need?

Retrieval answers questions whose answer is written down somewhere. Tool calls answer questions whose answer is computed by a system that already exists. Confusing the two is the most expensive category error in this field, because indexing a database as prose gives you a worse, slower, costlier version of a query you could already run.

The decision is per question type, not per project. Getting the split right usually removes more work than any optimisation you will do later.

The distinction that actually separates them

Not “structured versus unstructured” — that framing misleads, because plenty of prose lives in databases and plenty of tables live in documents. The distinction that holds is where the authoritative answer lives at the moment of asking.

If there is a system whose job is to know the answer — the orders service, the billing platform, the HR system, the ticket tracker — then that system is authoritative and anything you copy out of it is a stale replica. Call it.

If the answer is a sentence somebody wrote, in a policy, a manual, a contract, a design document, then the document is authoritative and retrieval is the mechanism for finding it.

Question	Authority	Mechanism
“What is our refund window?”	The policy document	Retrieval
“Can I still refund order 4471?”	The orders system	Tool call
“How much notice do I have to give?”	The contract	Retrieval
“How much leave do I have left?”	The HR system	Tool call
“What does error 4419 mean?”	The docs	Retrieval, keyword-first
“Why did my job fail last night?”	The logs or job runner	Tool call

Read those pairs left to right and the pattern is clear: the general rule is a document, the specific instance is a system. Most real user questions are a mixture — “can I refund this” needs the policy and the order — which is why routing matters more than choosing.

The five ways tool calls win

Correctness is inherited. The database is right by definition. You do not have to measure whether retrieval found the current value, because there is no retrieval and no copy.

Freshness is free. No index to update, no reindexing schedule, no window during which the system is confidently out of date. This eliminates most of the maintenance described in corpus drift and what it costs you.

Permissions are inherited too. If you call the API as the user, the API’s authorisation applies. Compare that with reproducing an access model inside an index and keeping it synchronised — which is where a large fraction of retrieval engineering effort goes.

Failure is legible. A wrong argument or a 404 is a debuggable event. A retrieval miss dressed as a confident answer is not.

“No such thing” is answerable. A query for a nonexistent order returns nothing. Similarity search always returns its nearest neighbours, which for a nonexistent identifier means something wrong and plausible. This is a genuine safety property, and it is the reason the identifier row of the table in when not to use RAG matters.

Where tool calls are the wrong reach

They are not universally better, and the cases where they fail are specific.

No API, or an API you cannot get. The system exists, the integration does not, and the team that owns it has a roadmap. Sometimes the honest answer is that the document describing the process is more accessible than the process.

The question does not decompose into parameters. “Is our approach to vendor risk consistent with our security policy?” is not a function call. Anything requiring judgement over prose is retrieval’s territory.

Too many possible calls. Model reliability at choosing among a small number of well-described tools is decent as of this writing; reliability at choosing among a hundred is not. Past a modest number of tools you need routing or grouping, and the complexity you added to avoid an index is now its own project.

Latency and rate limits belong to someone else. A tool call inherits the backing system’s performance, including its bad afternoons. Retrieval from your own index has a latency you control — one of the components accounted for in latency budgets for retrieval.

The cost comparison

The interesting difference is not per request; it is the fixed and standing terms.

Tool calls   = integration engineering per system, once
             + a small permanent share of API-change maintenance
             + inference tokens for the call and the answer

Retrieval    = ingestion + chunking + embedding + storage
             + permanent corpus and quality maintenance
             + inference tokens for a larger prompt

Two things fall out. First, tool calls have almost no standing cost beyond keeping up with API changes, while retrieval has a permanent maintenance obligation. Second, tool calls cost engineering per integrated system, so they scale badly in breadth and beautifully in depth — one system with fifty question types is cheap, fifty systems with one question type each is not.

Worked illustration, all inputs hypothetical. Suppose two weeks of engineering to integrate one internal API, at a loaded cost of, say, $8,000 — substitute yours. Against that, indexing the same system’s exported records as documents might take three days, but then carries an ongoing re-export, an index to keep synchronised, and a class of wrong answers that requires quality measurement to detect. Over eighteen months the integration is almost always cheaper, and it is definitely more correct. The exception is when the export is already happening for another reason.

Routing, which is the real design work

Once you accept that you need both, the design question is how a question gets to the right mechanism. Three approaches, in increasing order of cost and capability:

Let the model choose. Describe the tools, provide retrieval as one of them, and let the model decide. Simplest, works well when the tool set is small and the descriptions are sharp, and the failure mode is a wrong choice that you can see in the log.

Classify first. A cheap classification step routes to a mechanism before anything expensive happens. More predictable, adds a step to the latency budget, and gives you a place to enforce policy — such as always using the system of record for anything about money.

Hard-route by pattern. Anything matching an identifier format goes straight to the lookup. Crude, nearly free, extremely effective, and worth having in front of either of the above.

The recommendation

Enumerate your question types before choosing a mechanism, and mark each one with where its authority lives. This exercise takes an afternoon with a support log and routinely reveals that a third of the “RAG project” is a set of lookups.

Route anything with a system of record to a tool call, always. Correctness, freshness, and permissions all come free, and you avoid owning a replica of data somebody else maintains. Do not index a database because indexing is the thing you know how to do.

Use retrieval for the questions whose answers were written by a person. That is what it is for, and it is very good at it.

The threshold that flips it: when integrating the authoritative system would take longer than the useful life of the answer — a legacy platform being retired in six months, a team that cannot prioritise an endpoint this year — indexing its exported data is a defensible stopgap. Label it as one, give it an expiry, and revisit it, because a stopgap index is exactly the artefact that becomes permanent and starts producing confidently stale answers.