RAG explained: how AI finds evidence before it answers

Reviewed September 29, 2026. Examples are vendor-neutral; linked product documentation is used to explain specific retrieval techniques.

A language model can produce a plausible answer from patterns learned during training, but its internal parameters are a poor place to look up a current policy, a private manual, or the exact paragraph behind a claim. Retrieval-augmented generation, usually shortened to RAG, adds a search step: find relevant material first, then give that material to the model as context for the answer.

The 2020 paper that introduced the widely used term combined a generator's parametric memory with an external, non-parametric memory. Modern systems vary widely, but the durable idea remains the same: knowledge can live outside the model, where it is easier to update, inspect, and cite.

RAG has two pipelines

The preparation pipeline runs when content is added or updated. The answer pipeline runs for each question. A good interface makes this distinction visible because stale indexing and poor retrieval require different fixes.

Step 1: preserve document structure

Before creating embeddings, the system must extract useful text and metadata. A PDF may contain headings, tables, footnotes, page numbers, scanned images, or multiple columns. Flattening everything into one string can destroy the relationships a reader needs.

For a university handbook, useful metadata might include document title, academic year, chapter, page, audience, and effective date. These fields let the system filter obsolete editions and show where an answer came from.

Step 2: split text into retrievable chunks

A chunk should be small enough to retrieve precisely and large enough to keep its meaning. A fixed character count is simple, but it may separate a rule from its exception. Structure-aware chunking follows headings, paragraphs, table rows, or sections, then adds a small overlap where continuity matters.

There is no universal chunk size. Test with the questions users actually ask and record whether the necessary evidence appears in the retrieved set.

Step 3: retrieve by words, meaning, or both

Keyword search is strong when exact tokens matter: a regulation number, product code, person's name, date, or rare technical term. Vector search maps text into numerical embeddings and finds passages that are close in meaning even when they use different words.

Many production systems combine them. Microsoft documents hybrid search as running full-text and vector queries in parallel and merging their rankings with Reciprocal Rank Fusion. Its documentation also notes why the mix helps: vector retrieval finds conceptual similarity, while keyword search preserves precision for names, dates, and specialized terms.

Step 4: rerank, filter, and assemble context

Initial search usually favors recall: collect enough plausible candidates that the right passage is unlikely to be missed. A reranker then compares the query with each candidate more carefully. Metadata filters can remove the wrong department, language, permission group, or date range.

More context is not automatically better. The TACL paper Lost in the Middle found that model performance on multi-document question answering and key-value retrieval could drop when relevant information appeared in the middle of long inputs. The finding does not mean every current model behaves identically, but it is strong evidence against treating a large context window as a substitute for retrieval evaluation.

Context assembly should therefore remove duplicates, preserve source labels, place the strongest evidence clearly, and stay within a tested size.

Step 5: instruct the model to use evidence

The generation prompt should define what to do when evidence is sufficient, conflicting, or absent. A useful policy is:

Citations improve inspectability, but a citation can still be misplaced. The interface should let readers open the quoted passage, not merely display a document title.

A worked example

Suppose a student asks, “If I miss the final exam because I am ill, when must I submit evidence?” The collection contains three handbook editions and a faculty memo.

1. The query parser identifies final exam, illness, and a deadline.
2. A filter selects the current academic year and the student's faculty.
3. Hybrid search finds passages using “assessment,” “medical grounds,” and “supporting documentation,” even though the user's wording differs.
4. A reranker places the current memo above an older general handbook.
5. The model answers with the deadline, names the relevant document, and notes any exception.

If the current memo is missing from the index, generation cannot repair retrieval. The correct diagnostic is “freshness or ingestion failure,” not “the model needs a better personality.”

Where RAG fails

RAG grounds a model in retrieved content; it does not prove the content is true. A stale policy, biased dataset, or malicious document can still lead to a wrong answer. Source governance remains part of the system.

Evaluate retrieval and answers separately

Create a test set of realistic questions with expected evidence passages and acceptable answers. Include ambiguous wording, exact identifiers, dates, multi-part questions, unanswerable questions, and cases where two documents conflict.

• Retrieval recall: for how many questions does the candidate set contain the needed passage?
• Ranking quality: how high does that passage appear?
• Groundedness: are factual claims supported by retrieved evidence?
• Answer correctness: does the response resolve the question accurately?
• Citation quality: does each citation support the nearby claim?
• Abstention: does the system admit when the collection does not contain an answer?

Measure these after changes to parsing, chunking, embeddings, ranking, prompts, or models. A better final score without layer-level measures leaves you unable to explain why it improved.

When RAG is the right tool

Use RAG when answers must depend on a changing or private document collection and readers benefit from inspecting sources. It is less useful when the task is pure transformation of supplied text, when a structured database query can return the answer exactly, or when the source collection is too poorly governed to trust.

A careful RAG system behaves like a small research pipeline: it selects evidence, records provenance, distinguishes absence from uncertainty, and lets a reader verify the conclusion.