Roughly four-fifths of the effort in a good business chatbot goes into knowledge inventory, retrieval design and permissions. If a proposal skips those and leads with the model, it is describing a demo.
Start with an inventory, not a model
Before anything is built, list what the assistant will be expected to know and where each item currently lives. This exercise is unglamorous and it reliably determines the outcome of the project.
- What sources exist, and who owns each one
- Which are current, and how you can tell
- Which contain information not everyone should see
- What is only in someone's head and needs writing down first
- What contradicts something else, and which one wins
The last two are where projects lose weeks. A retrieval system cannot resolve a contradiction between two documents; it will confidently pick one.
Retrieval design decides answer quality
Once content is identified, three decisions matter more than prompt wording.
How content is split
Chunk on structure — headings, clauses, sections — rather than on a fixed character count. A policy clause split across two chunks becomes two half-answers, and the model will happily merge them into something that was never written.
How it is searched
Vector search alone misses exact terms: product codes, clause numbers, part identifiers. Combine it with keyword search and merge the results. Nearly every retrieval system that underperforms in production is vector-only searching content full of exact identifiers.
What permissions travel with it
Permissions attach at ingestion, so retrieval can filter before the model ever sees the text. Filtering afterwards is not a control; it is a hope.
Teach it to decline
The single highest-value behaviour in a business assistant is a clean refusal. When retrieval returns nothing above a relevance threshold, the correct output is that it does not know, plus a route to someone who does.
This has to be built and measured, not requested in a prompt. Set the threshold, test it against questions you know are unanswerable, and monitor how often it fires. A rate of zero means the threshold is wrong, not that the assistant is excellent.
Design the exit
Every assistant needs a way out that does not lose the conversation. Handoff should carry the transcript, the page the person was on and any details already captured into the helpdesk or CRM. Making the customer repeat themselves after a failed automated conversation is worse than not offering one.
The loop that keeps it accurate
Deployment is where the useful information starts. Two artifacts justify the effort:
- A gap report — questions where retrieval found nothing or the assistant declined. This is a content backlog written by your users.
- An evaluation set — a fixed list of real questions with acceptable answers, run before every change so you can tell whether quality moved.
Without the second one, every prompt change is a guess, and the third guess will quietly undo the first.
AI Chatbots
A chatbot is only worth deploying if it knows things the visitor could not find on their own — and admits when it does not.
AI Chatbots