One Index or Many?
One index with metadata filters and many separate indexes give you the same answers. They differ in what happens when something goes wrong, how much fixed overhead you carry, and how hard it is to prove that user A cannot see user B’s documents. That last item is the one that decides it more often than performance does.
The default should be one index with filters. Here is what should move you off the default.
What each shape actually is
One index, filtered. All chunks in one collection, each carrying metadata — tenant, team, sensitivity, source, language. Every query attaches a filter. Isolation is a property of your query-construction code.
Many indexes. A separate collection per tenant, team, or topic. Queries go to the relevant one. Isolation is a property of your routing, and the wrong result is unreachable rather than merely unselected.
The distinction is where isolation is enforced: inside a query, or by which endpoint you talk to. Everything else follows from that.
What splitting buys
Isolation you can demonstrate. With separate indexes, a bug in filter construction leaks nothing, because the other tenant’s data is not in the collection being searched. With one index, a missing filter returns everything. Both are correct when correct; only one fails safe. If you have to satisfy an auditor, “the data is in a different store” is a much shorter conversation than “our code always adds the filter.”
Precision, for free. Searching a small collection means fewer plausible-but-wrong neighbours to compete with the right chunk. Filtering achieves this too — the point is that a split gives it to you structurally rather than by remembering.
Independent lifecycle. Re-embed one tenant, migrate one team, delete one customer’s data by dropping their collection. That last one is a real operational advantage: deletion goes from a careful selective operation to a single destructive one you can verify.
Blast radius. A corrupted index affects one tenant. A bad bulk ingest affects one team. This matters most where tenants are external customers with contracts.
What splitting costs
Fixed overhead per index, times the number of indexes. Every collection carries some baseline memory and metadata. Small per index, ruinous at a thousand of them, and the fixed term is what makes per-tenant splitting infeasible at scale.
Cross-cutting queries get hard. “Search everything I have access to” becomes fan-out plus merge, and merging ranked lists from separate searches is not just concatenation — scores across collections are not necessarily comparable. You have introduced a distributed-systems problem in exchange for an isolation property.
Operational multiplication. Reindexing, backups, monitoring, and capacity planning all become per-index. Automation makes this manageable; the number of things that can be individually broken still went up.
Long-tail waste. In per-tenant designs, most tenants are tiny. You are paying fixed overhead for collections holding a handful of documents.
| Criterion | One index, filtered | Many indexes |
|---|---|---|
| Isolation strength | As good as your code | Structural |
| Fixed overhead | One baseline | Multiplied |
| Cross-cutting search | Natural | Fan-out and merge |
| Per-tenant deletion | Selective operation | Drop the collection |
| Operational surface | One thing | N things, automated |
| Small-tenant efficiency | Good | Wasteful |
The three reasons that justify a split
1. A contractual or regulatory isolation requirement. If a customer agreement or a regulator requires that their data be separately stored, the decision is made and cost is a detail. Do not argue this one on efficiency.
2. Genuinely different retrieval regimes. Separate collections are right when the content demands different treatment — a different embedding model per language, radically different chunk sizes for code versus prose, or one corpus that must be reindexed hourly while another is rebuilt monthly. Here the split reflects a real difference rather than an organisational one.
3. Volume large enough that partitioning is a performance necessity. At the scale where a single index no longer fits or no longer meets its latency budget, you are partitioning regardless. Prefer partitioning along a dimension that also gives you isolation, since you are paying for the split either way.
Not on the list, and commonly used as a reason: team ownership. That two teams own different documents is not a reason to give them different indexes, any more than it is a reason to give them different databases. Handle it with metadata and access control.
A middle shape that usually wins at scale
Per-tenant indexes fail on fixed overhead; a single index fails on demonstrable isolation. The pattern that resolves it is grouping: a small number of indexes, each holding many tenants, with filters inside.
Put the tenants that need genuine isolation — the large ones, the regulated ones, the ones whose contract says so — in their own collections. Put everyone else in a shared collection with filters. You get structural isolation exactly where you are asked to prove it, and one baseline overhead for the long tail.
The cost is that your routing layer now has two cases. That is a small amount of code in one place, and it is far cheaper than either extreme at scale.
The arithmetic
Worked illustration, all inputs hypothetical. Suppose fixed overhead per index is around 100 MB of memory, and memory costs you a hypothetical $5 per GB-month. For 500 tenants: 500 separate indexes is roughly 50 GB of pure overhead, about $250 a month before storing a single vector. Grouped into 10 shared collections plus 5 isolated large tenants: 15 indexes, 1.5 GB, about $8.
Substitute your own overhead figure — it varies substantially by store and index type, and you should measure yours rather than trust an estimate. The ratio is the finding: fixed overhead scales with your number of indexes and nothing else, so it is the term that punishes fine-grained splitting and the one nobody notices until the memory bill arrives. Sizing this properly is part of the exercise in what a RAG system actually costs to run.
Whichever you choose, filter anyway
Two habits that pay off regardless of shape.
Never build a retrieval call that can omit its filter. Make the authorisation context a required argument of your retrieval function, not an option a caller can forget. This is the single highest-value line of defence, and it is a code-structure decision rather than an infrastructure one.
Test the negative case. An automated test that asserts a user cannot retrieve a document they should not see, running on every commit. Permission bugs are the failures with real consequences, and they are invisible in ordinary usage because they only manifest for the wrong person — precisely the profile discussed in choosing an architecture by how it fails.
The recommendation
Default to one index with mandatory filters, and make the filter structurally impossible to omit. It is simpler, cheaper, and handles cross-cutting search naturally.
Split when isolation must be demonstrable, when retrieval regimes genuinely differ, or when volume forces partitioning. Three reasons, all of them concrete. Team boundaries are not one.
At multi-tenant scale, group rather than split per tenant. Isolated collections for the tenants who require it, shared collections with filters for the long tail.
The threshold that flips it: the first time someone outside your team asks you to prove that one customer’s documents cannot be returned to another, a filter-based answer stops being sufficient — not because it is wrong, but because it is not demonstrable in the form the question requires. That is the point to give that customer their own collection. Anticipating it is one of the permission thresholds in scaling thresholds for a retrieval system.