· 2 min read
Chatbots that answer from your own content: RAG in practice
What it takes to ship a client chatbot that stays on topic: chunking and indexing their documents, hybrid retrieval, prompts that cite sources and admit when they don't know, and how to test it before customers do.
Jose Rai
Senior software engineer
Most clients who ask for "a ChatGPT for our business" want the same thing: a chatbot that answers questions about their products, policies and docs, and doesn't make things up. The model is the easy part. The work is getting the right pieces of their content in front of it at the right moment. That pattern is retrieval-augmented generation, and this is how I build it.
Ingest: chunk by meaning, not by size
Splitting every document into fixed 500-token blocks is the most common mistake. It cuts tables in half and separates a heading from the paragraph that explains it. I split along the document's own structure, headings, sections and list items, and keep each chunk's title path and source URL as metadata. A chunk that knows it lives under "Shipping > International > Returns" is far easier to retrieve and to cite.
- Strip navigation, footers and cookie text before indexing. They add noise to every search.
- Deduplicate near-identical pages, or one policy repeated on ten pages will crowd out everything else.
- Store a content hash per chunk, so re-indexing only embeds what changed.
Retrieve: hybrid search, then rerank
Vector search is good at meaning and bad at exact terms like product codes, error messages and names. Keyword search is the opposite. I run both, merge the results, and rerank the top candidates against the question before choosing the few chunks that go into the prompt. That one step fixes most "the answer was in the docs but the bot didn't find it" reports.
export async function answer(question: string) {
// 1. Retrieve broadly from both indexes, then rerank to a short list.
const [semantic, keyword] = await Promise.all([
vectorIndex.search(await embed(question), { limit: 20 }),
keywordIndex.search(question, { limit: 20 }),
]);
const top = (await rerank(question, dedupe([...semantic, ...keyword]))).slice(0, 5);
// 2. Not enough relevant context: say so instead of guessing.
if (top.length === 0 || top[0].score < MIN_RELEVANCE) {
return { text: FALLBACK_MESSAGE, sources: [] };
}
// 3. Answer only from the numbered sources, and cite them.
const context = top.map((c, i) => "[" + (i + 1) + "] " + c.title + "\n" + c.text).join("\n\n");
const text = await llm.complete({ system: SYSTEM_PROMPT, user: context + "\n\nQuestion: " + question });
return { text, sources: top.map((c) => c.url) };
}Prompt for honesty, not just helpfulness
The system prompt tells the model to answer only from the numbered sources, to cite them, and to say it doesn't know when the sources don't cover the question. I also give it the client's tone and a short list of topics it should hand off to a human, like refunds over a certain amount or anything legal. The relevance threshold before the model is even called matters as much as the prompt: no context, no answer.
Test it like a product
Before launch I build a question set from the client's real support tickets and FAQs, plus questions the bot should decline. Each run checks three things: did retrieval find the right source, is the answer supported by that source, and did it refuse what it should refuse. Most fixes end up in chunking and retrieval, not in the prompt.
Run it
- Keep the index fresh. A scheduled re-crawl plus a webhook when the CMS publishes means the bot never quotes last month's price.
- Log questions and sources, with personal data removed. Unanswered questions are the client's best list of missing documentation.
- Give the client an admin view to see conversations, mark bad answers and add content. That turns the bot into something they can improve without me.
A good company chatbot is mostly a good search engine with a careful writer on top. Get retrieval right, make honesty the default, and test with real questions, and the model will look much smarter than it is.