Choosing a Freshness Requirement You Can Afford

Ask a stakeholder how fresh the index needs to be and the answer is “real time.” Ask what a wrong answer costs during a one-hour lag and the answer is usually “nothing much.” The gap between those two answers is a large amount of engineering, and closing it deliberately is one of the cheapest architecture decisions available.

Freshness is not one requirement. It is a per-document-type requirement, and most corpora have exactly one type that is urgent.

The four ingestion designs, priced by complexity

Each step up buys you latency and costs you a new failure mode.

Full periodic rebuild. Re-ingest and re-embed everything on a schedule. Staleness is up to one interval. Beautifully simple: no partial state, no synchronisation logic, and the rebuild is a smoke test of your whole pipeline every night. Cost scales with corpus size times frequency, so it becomes uneconomic for large corpora or short intervals.

Incremental batch. Periodically ingest only what changed. Staleness is still up to one interval, but cost is proportional to change volume rather than corpus size. New requirement: reliably knowing what changed, which is easy where sources have modification timestamps and awkward where they do not. New failure mode: a missed change is invisible, and the index quietly disagrees with reality until the next full rebuild — which you should still run occasionally, precisely for this reason.

Event-driven. The source system emits a change and you ingest it within seconds or minutes. Staleness is small. Requires the source to emit events, or a polling layer pretending it does, plus a queue, plus idempotent handling of duplicates and out-of-order arrivals. This is a real pipeline with real operational weight, and it is where a project’s complexity budget often quietly goes.

Read-through. Do not index the volatile thing at all; fetch it at query time with a tool call. Staleness is zero by construction. Costs a synchronous dependency and its latency, and it only works for facts a system can answer on demand — which is exactly the split in RAG or tool calls.

Design	Staleness	Cost driver	New failure mode
Full rebuild	Up to one interval	Corpus size × frequency	None significant
Incremental batch	Up to one interval	Change volume	Missed changes go unnoticed
Event-driven	Seconds to minutes	Engineering and operations	Queue lag, duplicates, ordering
Read-through	Zero	Per-request latency	Upstream dependency

Deriving the requirement from consequences

Do not ask how fresh people want it. Ask what happens if an answer reflects yesterday’s version, for each kind of document. The answers separate sharply.

Consequence: none. Reference material, background, how-to guides, architectural documentation. A day-old copy is indistinguishable from current. Nightly rebuild, and stop thinking about it.

Consequence: mild embarrassment. Product descriptions, org charts, process documents. Hours of lag are fine; a week is not. Nightly is still fine.

Consequence: a wrong commitment. Prices, entitlements, policy effective dates, availability. A stale answer here is a promise you did not intend to make. These need minutes at most — and usually should not be in the index at all, because a system of record already knows them.

Consequence: an access violation. A revoked permission, a document withdrawn for legal reasons, personal data subject to a deletion request. This is not a freshness requirement, it is a correctness requirement with an urgency attached, and it needs a synchronous path. Nightly is not an option no matter how convenient.

The insight worth acting on: rows one and two are the bulk of any real corpus, and rows three and four are usually a small, enumerable set. Which means the right architecture is almost never one freshness policy — it is a slow default with a fast path for a named minority.

The mixed design that is usually correct

1. Nightly full rebuild for the whole corpus. Simple, self-healing, catches everything you missed.
2. Incremental ingestion during the day for sources that change often enough to matter and expose modification times.
3. A synchronous deletion and revocation path that removes content from the index immediately, independent of the ingestion schedule.
4. Read-through tool calls for volatile facts, which are then never indexed and never stale.

Step three is the one to design first even though it feels like an edge case. Retrofitting immediate removal into a batch pipeline is unpleasant, and the requirement arrives without warning — a legal request, a security incident, an employee departure. It is one of the two thresholds worth pre-empting in scaling thresholds for a retrieval system.

Pricing the difference

Rebuild-only  = (corpus tokens × embedding price) × rebuilds per month
              + a small amount of scheduling engineering

Event-driven  = change volume × embedding price
              + queue infrastructure
              + engineering to build it
              + a permanent share of operational attention

Worked illustration, all inputs hypothetical. A 6-million-token corpus at a hypothetical $0.02 per million tokens costs about $0.12 to embed. Rebuilt nightly, that is roughly $3.60 a month. Substitute your own price; the conclusion is likely to survive.

Meanwhile an event-driven pipeline is perhaps three weeks of engineering at a hypothetical $4,000 a week — $12,000 — plus queue infrastructure and a permanent share of on-call. The embedding compute is not the cost of freshness. Engineering and operations are the cost of freshness, which is why the right question is never “can we afford to re-embed more often” but “what does this pipeline commit us to maintaining.”

Note the asymmetry: the rebuild cost grows with corpus size, so at very large corpora the arithmetic genuinely does force incremental ingestion. That is a threshold you can compute rather than guess.

What to write down

Freshness is a requirement, so record it like one — a short table in the design document with three columns: document type, maximum acceptable staleness, and the consequence of exceeding it. Third column is not optional. It is what stops the requirement inflating in a later meeting, and it is what justifies the engineering when the fast path genuinely is needed.

The same table tells you what to monitor. If nightly rebuild is your policy, an alert when a rebuild has not completed within a day is the entire freshness monitoring requirement, and it is cheap.

The recommendation

Default to a nightly full rebuild and defend it. It is the simplest correct design, it self-heals, and for most corpora the compute is immaterial. Justify anything more complex against a named consequence.

Build the synchronous removal path from day one, separately from ingestion. Deletion and revocation are not freshness, they are correctness, and they arrive urgently.

Move volatile facts out of the index rather than making the index faster. A price that changes hourly should be a tool call, not an ingestion problem. This removes the requirement instead of paying for it.

The threshold that flips it: when a full rebuild no longer fits comfortably in its window — because the corpus grew, not because someone wants fresher answers — move to incremental, and keep a weekly full rebuild as your reconciliation mechanism. Wanting fresher answers, on its own, is not a reason; a named consequence of staleness is. And if the consequences you name are severe and the material is volatile, reconsider whether retrieval is the mechanism at all: when not to use RAG.