The Assistant You Already Pay For
Build-or-buy is usually argued as two options, and there is nearly always a third: the document assistant bundled with software your organisation already licenses. It is free at the margin, requires no engineering, and is frequently good enough for the use case being proposed. It is also the option nobody in the room is incentivised to raise.
Evaluate it first — not because it usually wins, but because knowing why it loses is how you justify the budget for anything else.
Why this option gets skipped
Three predictable reasons, and it is worth naming them out loud so the discussion is honest.
Nobody in the meeting is excited by it. Engineers want to build; vendors want to sell; the sponsor wants a story. “Turn on the feature we already have” is nobody’s preferred outcome.
It is invisible on the org chart. The suite is owned by IT or procurement, the assistant idea is owned by a product or engineering team, and neither is watching the other’s roadmap.
It looks unimpressive in a demo comparison. A bundled assistant over your real, messy corpus will demo worse than a hand-built prototype over five clean documents. That comparison is not measuring what it appears to measure — it is measuring corpus preparation, which you will have to do either way.
What bundled assistants tend to be good and bad at
Stated structurally, because specific product capabilities change constantly and any list of them is wrong within months. As of this writing the shape holds even as the details move.
Structurally good at: authentication and permissions, because they already know who the user is and what they can open; breadth across the file types their own suite produces; zero deployment; and staying current with sources that live inside that suite.
Structurally bad at: anything outside their ecosystem; how documents are chunked or retrieved, which you cannot reach; measurable quality, since you get no eval hooks and no visibility into what was retrieved; and any behaviour you want to change, since the answer to “can it do X” is set by their roadmap.
That last pair is the real trade. You are exchanging all control over quality for all of the integration work. Whether that is a good trade depends entirely on whether your differentiation lives in the answers.
| Criterion | Bundled assistant | Something you build |
|---|---|---|
| Time to first useful answer | Days | Weeks to months |
| Marginal cost | Often zero or a per-seat uplift | Infrastructure plus permanent staffing |
| Permissions | Inherited, correct by construction | Your problem, and it is the hard part |
| Sources outside the suite | Limited or absent | Whatever you integrate |
| Retrieval quality control | None | Total |
| Quality measurement | Little to none | Whatever you build |
| Embedding it in your product | No | Yes |
The four questions that decide it
1. Is the audience internal staff, or your customers? A bundled assistant answers your employees inside your suite’s interface. It generally cannot be a feature of the product you sell. If the requirement is customer-facing, this option is out in one sentence — which is a fast and useful outcome.
2. Do the documents already live in that suite? If the corpus is the same wiki, drive, and mail the assistant already indexes, you are getting ingestion, permissions, and freshness for nothing — collectively the most expensive parts of what you were about to build. If half the corpus lives in a legacy system, you get half a product, and half a product is often worse than none because users cannot tell which half they are in.
3. Does anyone need to tune the answers? If the answers are the product, you will need to reach the ranking behaviour within months, and a closed assistant becomes an obstacle. If it is a convenience for finding internal documents, defaults are probably better than what you would tune in the same time.
4. Can you measure whether it is working? Usually not, beyond asking users. For a convenience tool that is acceptable. For anything where a wrong answer has consequences, the absence of measurement is disqualifying — see choosing an architecture by how it fails.
The arithmetic, which is unusually one-sided
Worked illustration, all inputs hypothetical. Suppose the bundled assistant costs a per-seat uplift of, say, $20 a month for 200 staff: $4,000 a month. Against a build: infrastructure of a few hundred a month, plus the permanent fractional staffing from the team you need to run retrieval in production — call it 35% of an engineer at a hypothetical $16,000 loaded, so $5,600 — plus inference. Substitute your own figures throughout.
Even in that hypothetical the numbers are close, and they are close before counting the build cost, which is the part everyone estimates and the part that is genuinely one-off. Notice what drives the comparison: seat count on one side, headcount on the other. Small staff means the bundled option is cheap; large staff means it can easily be more expensive than building. That is the reverse of most people’s intuition, and it mirrors the asymmetry in build or buy your retrieval stack — buying scales with usage, building scales with headcount.
The evaluation that takes a week
Do this before approving any build, and make it fair rather than favourable:
- Point it at the real corpus, not a curated subset. Messy is the condition you are actually buying for.
- Use real questions — twenty or thirty from a support queue or a survey of the intended users, written down before anyone sees an answer.
- Have two people score each answer as useful, partly useful, or wrong, with the wrong ones categorised: missing source, stale source, retrieved the wrong thing, or answered from nothing.
- Note the permission behaviour explicitly. Ask a question whose answer sits in a document the asker cannot open, and check what happens.
- Write down the failure categories. If most failures are “the source was not in the suite” you have learned that ingestion is your problem, which is information you needed regardless.
That week is the cheapest requirements-gathering available, and its output is reusable: the question set becomes the beginning of an eval set for whatever you build instead.
The recommendation
Run the week-long evaluation before funding a build, and treat its output as a requirements document rather than a verdict. The most valuable result is not whether the assistant was adequate — it is the failure categories, which tell you what you would actually be building.
Take the bundled assistant when the audience is internal, the corpus lives inside the suite, and nobody needs to tune the answers. All three conditions, not two. This describes a large share of “we need an internal knowledge chatbot” requests, and satisfying one with a licence you already hold is a genuinely good outcome.
Build when the assistant is customer-facing, when the corpus spans systems the suite cannot see, or when answer quality is your product. Any one of those is sufficient.
The threshold that flips it: when more than about a third of the questions you care about fail because the source material lives outside the suite, the bundled option cannot be fixed by configuration, and the ingestion work you would need is most of a retrieval project anyway. At that point build — and use the failure list as your first backlog.