RAG explained: giving AI the right documents before it answers

The basic sequence

Imagine a service assistant answering a warranty question. A retrieval system searches the approved manuals or policies, selects relevant passages, and supplies them with the question. The model uses those passages to draft an answer. IBM's overview explains this combination of retrieval and generation. IBM: What Is Retrieval-Augmented Generation (RAG)?

The documents may be divided into smaller chunks and indexed for search. Embeddings can help retrieve passages with similar meaning even when the wording differs. Keyword search can still be useful, especially for exact part numbers, names, and dates. A practical system may combine several retrieval methods.

Why it can fail

The correct document may not be indexed. The wrong edition may rank first. A passage may be split away from its exception. The user may ask a question that no source answers. Even if retrieval succeeds, the model may misread the passage or add unsupported details.

Evaluate those stages separately. For a failed answer, ask whether the necessary evidence was retrieved and whether the answer faithfully used it. Otherwise, you may keep changing the model when the real problem is document quality or search configuration.

A small design exercise

Prepare a few short fictional policy documents with clear titles and dates. Include one outdated version, one exception, and one question the documents cannot answer. Define the expected source and answer for each test question before running the system.

Answer using the retrieved policy excerpts only. Cite the document title and section for each important statement. If the excerpts disagree, show the disagreement. If they do not establish the answer, say what additional document is needed.

The request sets a useful target, but the application still needs reliable retrieval and checking. A prompt cannot supply an access boundary that the underlying system fails to enforce.

What matters in a real deployment

Keep source ownership, update dates, and deletion procedures clear. Enforce permissions before retrieval exposes text to a model. Treat retrieved documents as information, not as instructions that can override the user's task. Log enough information to investigate mistakes without collecting unnecessary sensitive content.

For evaluation, track whether the right passages were found, whether citations support the answer, and how the system handles unknowns. A well-maintained small collection can be more useful than a vast pile of inconsistent documents. RAG is an information-management project as much as a model feature.