RAG, Fully Explained
How an internal document chatbot is actually built
① RAG (retrieval-augmented generation) doesn't train new knowledge into a model — it searches for relevant documents each time a question comes in and hands them over as reference. The model itself never changes.
② That's why updating the documents alone reflects instantly — no retraining needed. That's why RAG is the standard for internal document chatbots.
③ But the accuracy of the answer ultimately depends on whether the search found the right document. Feed it the wrong snippet, and the model builds a plausible, wrong answer on top of it.
① Why RAG is needed — the knowledge cutoff and internal documents
A model fails to answer for one of two reasons. One is the knowledge cutoff — it never learned about anything after training ended. The other is content that was never in the training data to begin with — a company's internal documents, personal files, anything never made public. Both boil down to the same thing: "the model doesn't know it."
"So why not just fine-tune on our company documents?" is a natural thought, but as covered in how AI is made, it doesn't work well. Fine-tuning data is tiny compared to pre-training, so it can't reliably implant new facts, and knowledge stuffed in clumsily becomes fuel for hallucination. On top of that, if a document changes even once a day and you'd have to retrain every time, the cost and time don't add up.
RAG changes the approach itself. Instead of expanding what's inside the model's head, it finds the document you need each time you ask and sets it beside the model. The model answers by looking, not by remembering.
② How it works — four steps of retrieval
The name says it — retrieval + augmentation + generation. In practice it runs in this order.
| Step | What happens |
|---|---|
| ① Chunking | A long document is split into small pieces (chunks) at the paragraph or section level |
| ② Embedding | Each chunk is converted into number coordinates called an embedding and stored in a vector database |
| ③ Retrieval | When a question arrives, it's embedded the same way, and the chunks with the nearest coordinates are found |
| ④ Generation | The retrieved chunks are attached to the prompt as "reference material" and sent to the model → the model answers grounded in that material |
The embedding in step ② is exactly the same concept covered in how an LLM works — it directly reuses the property that text with similar meaning sits close together in coordinate space. A query like "vacation policy" and a document titled "annual leave guide" are worded differently but land near each other in coordinates, which is how the search finds it. RAG isn't so much a new technology as applying an embedding property that already existed to search.
Adding RAG doesn't change a single parameter in the model. Only the reference material attached ahead of the prompt changes each time. So adding or editing a document is reflected starting with the very next question — no waiting for retraining.
③ Why hallucination drops — and why it doesn't disappear
As covered in the hallucination guide, hallucination happens because the model generates high-probability sentences without checking facts. RAG cuts this sharply by making the model answer from a document in front of it instead of pulling from memory — there's material to copy, so there's no need to make anything up.
But it doesn't go away entirely. It can still go wrong in three places.
- When retrieval pulls the wrong chunk — close in coordinates but actually unrelated to the question. The model builds a plausible answer on top of it anyway.
- When retrieved chunks contradict each other — if both an old policy and a new one get retrieved, there's no guarantee which one the model prioritizes.
- When the answer isn't in any document — even with no results or poor ones, the model can still risk inventing a plausible-sounding answer.
That's why it matters to require a RAG chatbot to show its source (which document, which section) alongside the answer. The "line that demands reasoning" from prompt templates becomes the exact verification tool here — open the cited source and you can check right away whether the retrieval was right.
④ Why chunk size matters
How finely you split a document sounds trivial but changes the outcome a lot.
| Chunking | The problem |
|---|---|
| Too small | Retrieval is precise but context gets cut, so the chunk alone doesn't make sense ("refunds are not allowed" survives while "except within 7 days" is severed into a different chunk) |
| Too large | Unrelated content rides along too, wasting prompt space and burying the part that's actually needed |
In practice, splitting at the paragraph or section level, with a bit of overlap between adjacent chunks, is common to reduce broken context. There's no perfect answer here — it's tuned to the document's nature.
⑤ RAG vs. fine-tuning vs. just pasting everything in — which one, when
| Method | When to use it | Downside |
|---|---|---|
| Paste it all into the prompt | A handful of documents, a one-off question | More documents hit context limits and cost |
| RAG | Many documents that keep growing, where freshness matters (the standard for internal chatbots) | Accuracy is bounded by retrieval quality |
| Fine-tuning | Changing tone, format, or behavior, not knowledge | A poor fit for keeping knowledge current |
The three aren't rivals — they're often combined. Fine-tune the answer format, and attach the latest internal information with RAG.
⑥ Security — RAG is a standard prompt-injection target
RAG's own structure can become the vulnerability. As in the scenario covered in the prompt injection guide, an attack succeeds just by planting an instruction inside one of the documents the search can retrieve. If the chatbot is told "answer based on the following document," and that document has a sentence hidden in it like "ignore the previous instructions…," it risks mistaking that for a real instruction.
If you build or run a RAG chatbot, who can add or edit the documents it searches is the security boundary, full stop. Whether it's an insider or a file that drifted in from outside, anything that lands in the document store without verification is a potential attack vector.
⑦ A checklist — evaluating or building an internal document chatbot
| Check | Why it matters |
|---|---|
| Does the answer show its source (document, section)? | Lets you verify right away whether retrieval was correct |
| Is an edited document reflected starting with the next question? | If it needs retraining, it may not actually be RAG |
| Does it say "I don't know" when nothing relevant turns up? | Confidently inventing an answer instead is a red flag |
| Is what's searchable separated by access level? | HR or payroll documents shouldn't be retrievable by an all-staff chatbot |
| Is there a vetting step for document uploads? | Without one, it's wide open to prompt injection |
Companies rapidly adopting RAG-backed tools over internal documents and customer data is one real face of the enterprise-AI demand story that keeps showing up in recent Big Tech earnings (the Big Tech earnings guide). The retrieval and embedding computation itself also adds to inference compute, pointing the same direction as the shift in data-center demand from training to inference (the semiconductor sector deep dive).
RAG isn't a technology that makes a model smarter — it's a search system that finds the right reference material for each question and sets it beside the model. The model just answers from what it's shown. Which means the accuracy of the result ultimately depends on how accurate the search is, and showing the source is the only window you have to check that.
※ Written as of September 2026, describing general principles. The actual implementation (chunk size, retrieval algorithm, vector database choice, and so on) varies by service and tool; this piece covers the conceptual flow.
※ This guide is provided for general educational purposes and simplifies technical details for readability.
New guides, when they land
We publish AI literacy guides twice a week. Subscribe and the next one comes to you — free, unsubscribe anytime.