RAG Without Hallucinations: Chunking, Retrieval and What Drives Answer Quality

Reading time: approx. 8 minutes

Short answer: RAG hallucinates less than a bare model, because it answers from your documents, but it does not stop hallucinating entirely unless you design three things well: how you split documents into fragments (chunking), how effective the search is (retrieval), and honest behavior when there is no basis. The most common cause of wrong answers is not a "dumb model" but the fact that the right fragment was not found, or that a fragment similar in wording but off-target was found. Below, step by step, what really drives quality.

Where errors in RAG come from

A text-generating model always tries to produce a coherent, plausible-sounding answer. If the search system feeds it good, relevant fragments, the answer will be grounded in facts. If it feeds it loosely related fragments, or nothing, the model may stitch together something that sounds good and is wrong. That is why in RAG the fight for quality happens first at the search stage, before the model even starts writing.

Chunking: how you split documents

Documents are not fed into the system whole. They are split into fragments that are then searched. Against intuition, this is one of the most important decisions.

• Fragments that are too large mix several topics at once, so search loses precision and the model gets a lot of noise.
• Fragments that are too small lose context. A parameter alone, without which machine and which procedure it belongs to, is useless.
• A good split follows the structure of the document: a section, a procedure, a table as a whole. In technical documentation it is worth keeping headings and the fragment's link to a specific machine or version.

Practical rule: split by meaning, not by a fixed character count. A fragment should be a self-sufficient answer to a single question.

Retrieval: how the system finds the right fragment

Search in RAG usually relies on embeddings, that is, turning text into a representation that lets you search by meaning rather than exact words. Thanks to that, the question "how do I clear the lockout" will find a fragment about "resetting the alarm", even without shared words.

This works well, but has limits. Abbreviations, catalog codes and shop-floor jargon can be tricky, because they look similar to one another. So two approaches are often combined: search by meaning and classic search by keywords. The first catches paraphrases, the second catches exact symbols and part numbers.

Reranking: when it helps and when it only costs

Reranking is an extra step where a separate model reviews the found fragments and orders them from most relevant. It can noticeably improve quality, but is not always worth its price.

• It helps when the base is large and noisy and the first search returns many "almost relevant" fragments. Reranking separates the wheat from the chaff.
• It is redundant when the base is small and well chunked, and search already returns the right fragment at the top. Then it only adds latency and cost.

Conclusion: do not add reranking by default. Add it when tests show the first search loses relevant fragments in the noise.

Designing the "I don't know" answer

This is the most underrated element of quality. A system that can say "I found no basis in the documentation" is, in technical work, more valuable than one that always answers. A blank field is information: either a document is missing or the question is poorly posed. A confident, wrong answer is a cost, because someone may act on it. Enforcing source citation and allowing "I don't know" are the two decisions that most limit hallucinations.

How to measure quality before you trust it

You will not trust a system on gut feeling. A simple, cheap way to evaluate:

1. Collect a set of real questions to which you know the correct answers and know which document they are in.
2. Check two things separately: whether the system found the right fragment (search relevance) and whether the answer is correct (answer relevance).
3. Record the cases where it got it wrong, and look at which stage failed: it did not find the fragment, or it found it but answered badly.

That separation is key, because you fix different things. Bad retrieval is fixed with chunking, better search, or reranking. A bad answer given a good fragment is a matter of the model and how the answer is framed.

The most common mistakes

• Loading raw scans without OCR. The system cannot see the content, so it will not find the answer.
• No version tagging. The system mixes the old and new procedure.
• Chunking by fixed character count. Fragments cut sentences and tables in half.
• No "I don't know" process. The system always answers, so it cannot be trusted.
• No evaluation. Nobody knows how well it works, so quality quietly degrades.

What this post doesn't cover

I don't get into choosing a specific embedding model or into infrastructure architecture. I also don't cover where to source knowledge and how to maintain it over time, since that is a separate topic on service knowledge management. The focus here was one thing: why RAG gets it wrong and which design decisions most limit hallucinations.

Related notes

• RAG for Technical Documentation: How AI Uses Your Manuals and Service Records
• AI for Service Knowledge Management: Feeding the System and Its Limits