Many wrong answers from a RAG chatbot come from its search step, not the model. If the right piece of your documents never reaches the prompt, the model has nothing to work from, and you can test that step on its own.
RAG (retrieval augmented generation) works in three moves. Split your documents into chunks, slices of a few hundred tokens each (a token is roughly a word fragment). Find the few chunks closest in meaning to the question. Paste them into the prompt so the model answers from them.
"Closest in meaning" comes from embeddings. Each chunk is turned into a list of numbers that places its meaning at a point in space, so similar text lands close together.
The weak spot is that a chunk is cut off from its document. A slice that says "this was removed in version 3" no longer says what "this" is.
So when an answer is wrong, don't start by editing the prompt. Write about 50 real questions and label each one with the chunk that holds its answer. Then check the search alone: did that chunk show up in the top 20 results? If it did and the answer is still wrong, the problem is generation, the part where the model writes its reply. If it didn't, no prompt change will help, because the model never saw the facts.
One more rule of thumb: if your whole knowledge base is under about 200,000 tokens, you may not need RAG at all. Paste all of it into the prompt.
Quick check before you scroll: A RAG chatbot keeps giving wrong answers about one company's quarterly results, even though the correct report is indexed. What is a likely cause, and what is one fix?
Full breakdown + the answer: frankduah.me/learnings/2026-10-31-rag-architecture-retrieval-augmented-generation
The answer
The chunk containing the answer probably lacks context (no company name or quarter), so search ranks it low and it never reaches the prompt. Adding a short generated description to each chunk before embedding, or combining keyword search with a reranker, helps.
Before you open the breakdown, put your guess in the comments: why would that chatbot miss a quarterly report it already has indexed, and what's one fix you'd try first?
New here? I post a bite-size AI / ML concept like this every day. Follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#RAG #GenerativeAI #AI #LLM #AIAgents #MachineLearning