Self-Hosting Your Embedding Model: When It Pays
Embedding is the one part of a RAG pipeline where self-hosting is genuinely easy: the models are small, open-weight options are strong, and inference is a single forward pass with no generation loop. It is also the part with the smallest bill, which means the cost argument for self-hosting is usually weak and the non-cost arguments are usually the real ones.
Run the arithmetic anyway, because it takes ten minutes and it settles the discussion.
Why the bill is small
Embedding cost scales with tokens processed, and you process two very different volumes.
Ingestion is a one-off per document: your whole corpus, once, plus changes as they arrive. Even a large corpus is a bounded number of tokens, and it is charged once rather than per query.
Query embedding is per request, but a query is short — a handful of tokens against thousands in the generated answer’s prompt. As a share of per-request cost it is nearly always negligible next to the generation step.
So the total is dominated by a one-off ingestion charge and a permanent trickle. Compare that against a GPU instance, which is a standing hourly cost whether or not anything is being embedded, and the shape of the comparison is clear before any numbers go in: hosted APIs are cheap for bursty, low-volume embedding; self-hosting needs sustained volume to beat idle capacity.
The arithmetic
Hosted = tokens embedded × price per token
(corpus once, plus updates, plus one query embedding per request)
Self-hosted = instance hours × hourly rate
+ engineering to deploy and maintain the service
+ a share of on-callWorked illustration, all inputs hypothetical — substitute your own. Take a 6-million-token corpus and 2,000 queries a day at 20 tokens each, so 40,000 query tokens daily, about 1.2 million a month.
Hosted, at a hypothetical $0.02 per million tokens: the corpus is $0.12 once, and queries are about $0.02 a month. The bill is effectively zero.
Self-hosted, on a modest GPU instance at a hypothetical $0.50 an hour running continuously: about $360 a month, plus deployment engineering, plus the service being one more thing that can page someone.
At this scale the hosted API wins by three orders of magnitude, and no amount of engineering cleverness closes that. It is worth stating plainly because the instinct to self-host for cost reasons is common and, at typical volumes, simply wrong.
Where does it flip? You need enough sustained token throughput that the hosted charge exceeds the instance cost. At the hypothetical prices above, $360 a month of hosted embedding is 18 billion tokens — a corpus far larger than most organisations have, re-embedded frequently. Run the same division with your own prices; the conclusion is usually that you are nowhere near, and that is useful to know.
Two realistic exceptions. Continuous large-scale ingestion — millions of new documents a month — genuinely gets there. And repeated full re-embeds, if you are experimenting with chunking or models weekly, can add up, though the fix for that is usually to test on a sample rather than to buy a GPU.
The three reasons that actually decide it
Since cost rarely does, these are what the decision comes down to.
1. Data cannot leave your infrastructure. If embedding your documents means sending their text to a third party, and a residency or contractual requirement forbids that, the arithmetic is irrelevant. This is the most common legitimate driver, and it is a constraint rather than a trade-off — see data residency as an architecture constraint.
2. You want immunity from deprecation. A hosted model can be retired on the provider’s schedule, and that forces a full re-embed and re-validation on their timeline rather than yours, per embedding model lock-in and what switching costs. A model file on your own storage cannot be retired by anyone. If your corpus is large enough that a forced re-embed is a real disruption, this is a serious argument.
3. You need a domain-specific or fine-tuned embedding model. If a general model collapses distinctions your vocabulary depends on and you intend to adapt one, you are hosting it — nobody else will. Note that this is a substantial undertaking and the cheaper first move is usually a keyword layer alongside dense retrieval.
Not on the list: latency. A hosted embedding call is one network round trip, tens of milliseconds, and it happens once per request. It is rarely the problem — check the accounting in latency budgets for retrieval before optimising it. Self-hosting can shave that hop, but if your budget is tight enough for that to matter, the generation step is where the seconds are.
What self-hosting commits you to
If you go this way, price the whole commitment rather than the instance.
| Obligation | Notes |
|---|---|
| A service to deploy and version | Ordinary work, but permanent |
| Capacity for ingestion spikes | Re-embedding a corpus is a burst; either it is slow or you scale |
| Consistency between ingest and query | The same model and version at both ends, or your vectors are silently mismatched |
| On-call | Embedding down means ingestion stopped and queries fail |
| Model file custody | Storage, provenance, and reproducibility of exactly which weights you used |
Row three is the one that bites in practice: a model version drifting between the ingestion path and the query path produces a subtly broken index with no error anywhere. Pin the version explicitly in both places and record it with every vector.
The middle option
You can self-host the ingestion path and use a hosted API for queries, or the reverse, only if both use the identical model and version — the same open-weight model, served by you in batch and by a provider online. This is occasionally the right answer for the deprecation-immunity case, since you hold the weights but do not carry the online serving burden. It doubles your version-consistency risk, so do it deliberately and verify with a test that embeds the same text through both paths and compares.
The recommendation
Use a hosted embedding API unless a residency requirement, deprecation exposure, or a domain-specific model forces otherwise. At typical volumes the cost difference favours hosted by orders of magnitude, and the operational surface is one fewer service.
Choose an open-weight model even when you use it through a hosted provider, where that option exists. It costs nothing, and it means the deprecation escape hatch — serving the same weights yourself — stays available without a re-embed.
If you self-host, pin the model version in both the ingestion and query paths, and record it alongside every vector. This is the failure that is hardest to notice and easiest to prevent.
The threshold that flips it: divide your monthly hosted embedding spend by the monthly cost of an instance that could serve it. Until that ratio is comfortably above one — and for most teams it is a small fraction — self-hosting for cost reasons is a hobby, and the honest reasons to do it are the three named above. If you are running that division for the first time, do the same for the whole system: what a RAG system actually costs to run.