Do You Need a Vector Database at All?
Adding a dedicated vector database is often the first infrastructure decision a RAG project makes, and it is frequently premature. Embeddings can live in a file, in the database you already run, or in a search engine you already operate — and for corpora below a certain size all four options return the same results, differing only in what you have to operate.
The decision is about operational surface, not capability. Here is how to size it.
The four places embeddings can live
In memory, from a file. Load vectors at startup, compare against all of them per query. No service, no operator, no network hop. For small corpora this is not a hack — it is the option with the fewest failure modes, and exhaustive comparison over a modest number of vectors is fast.
In your existing relational database. Most mainstream databases now support a vector column and an approximate index, as of this writing. You inherit your backups, your migrations, your monitoring, your access control, and the ability to filter by ordinary columns in the same query — which is a bigger advantage than it sounds, because metadata filtering is what keeps precision up as a corpus grows.
In a search engine you already run. If you already operate a full-text search cluster, it probably does vector search too, and it definitely does keyword search well. That combination in one system is convenient, because hybrid retrieval is a common destination.
In a dedicated vector store, self-hosted or managed. Purpose-built: index types tuned for recall against latency, quantization to cut memory, filtering designed for vector workloads, and horizontal scaling. Real advantages that begin to matter at scale, and a new thing to operate — or a new invoice — before they do.
What the dedicated option actually buys
Be precise, because the marketing is not.
It buys performance at scale. Different index families make different recall-latency-memory trades, and a dedicated store gives you those knobs. Below a few hundred thousand vectors, few teams need them.
It buys memory efficiency. Quantization and tiered storage matter when the index no longer fits comfortably in RAM. Below that point the saving is theoretical.
It buys vector-aware filtering. Combining a similarity search with metadata constraints efficiently is genuinely harder than it looks, and general-purpose systems handle it less gracefully at large scale.
It does not buy retrieval quality. This is the misconception worth correcting. Your results depend on the embedding model, chunking, and query handling. A dedicated store returns approximately the same neighbours the naive option would, sometimes slightly fewer of them, since approximate search trades recall for speed. Nobody’s answers improved by moving to a vector database.
The sizing table
| Corpus scale | Reasonable choice |
|---|---|
| Thousands of chunks | In-memory from a file |
| Tens of thousands | Your existing relational database |
| Hundreds of thousands | Existing database or search engine; measure before moving |
| Millions and up, or latency-critical | Dedicated store, probably managed |
| Any scale, with a hard latency target under a search engine’s floor | Dedicated store |
| Any scale, with an existing full-text cluster and hybrid plans | The search engine you already run |
The rows people skip are the first two, and skipping them is how a project acquires a service to operate before it has a user.
The cost that is not the price
The interesting cost of a dedicated store is not its bill; it is that it adds a stateful system to your operational surface. Stateful means backups you have tested, a restore procedure someone has rehearsed, upgrades that are not trivial, and capacity you have to plan.
Cost of a dedicated store =
subscription or infrastructure
+ integration engineering
+ a permanent share of on-call attention, if self-hosted
+ one more thing that can be the cause of an incidentWorked illustration, all inputs hypothetical. Suppose a managed vector store quotes $300 a month. Against that, adding a vector column to a database you already run costs nothing incremental in operations — you are already backing it up and already on call for it. The real comparison is $300 a month plus an integration against zero plus a schema migration. At small scale the existing database wins easily, and it keeps winning until you hit a limit you can measure.
The corollary applies to self-hosting too: a self-hosted vector store is not $300 a month of engineering attention, it is a standing on-call commitment. That is the same asymmetry as in build or buy your retrieval stack — if you have decided you need a dedicated store, buy it managed unless residency or volume says otherwise.
Three signals that you have outgrown the simple option
Not opinions — things you can measure, which is what makes this a threshold rather than a preference.
Query latency at the retrieval step exceeds your budget for it. Measure it. If retrieval takes 40 ms and your budget allowed 200, the store is not your problem; see latency budgets for retrieval for where the time actually goes.
The index no longer fits in available memory, or your database is being squeezed by it. This is a real and legible signal, and it is the most common honest reason to move.
Filtered searches have become slow or awkward. As metadata filtering becomes central — which happens as corpora grow heterogeneous, per scaling thresholds for a retrieval system — general-purpose engines can struggle in a way purpose-built ones do not.
Absent all three, moving is speculative work.
Migration is cheap, which is the point
The reason to start simple is that the switch is genuinely low-cost, provided one condition holds: your source documents and chunking logic live outside the store. If you can rebuild the whole index from originals with one command, changing stores is re-running ingestion against a new target — days, not a project.
That single property makes almost every store decision reversible, and it is the same property that makes embedding-model changes affordable, per embedding model lock-in and what switching costs. Protect it and the rest of the infrastructure choices stay cheap.
The recommendation
Start with the database you already run. For the overwhelming majority of projects this is the right answer for the whole first year and often permanently. You get transactional consistency between documents and vectors, filtering in the same query, and no new operator — and you can stop having the vector database conversation entirely.
Start in memory if the corpus is small and static enough, and do not apologise for it. Fewer moving parts is a feature, and it will outperform a network round trip.
Buy a managed dedicated store when you have measured one of the three signals, not before, and buy rather than self-host unless a residency requirement or genuine volume says otherwise.
Keep ingestion reproducible from source documents so this decision stays reversible. That discipline is worth more than picking correctly the first time.
The threshold that flips it: when your vector index becomes the reason your primary database needs to be bigger, or when measured retrieval latency exceeds its share of your response budget, move. Both are numbers you can watch, and until one of them moves, the dedicated store is a solution to a problem you do not yet have.