RAG stands for retrieval-augmented generation. It combines a search step with a language model: find relevant information, include it in the model's input, and ask the model to answer using that information.
Why add search? Suppose your organization changes its cancellation policy this morning. The model's training does not automatically include that change. You could paste the new policy into a prompt, but what happens when you have thousands of documents? RAG helps select the passages to include.
Before you begin: Understand embeddings, model context, and source-grounded answering.
How retrieval and generation work together
The search component is called a retriever. It selects passages that may help answer the question. The application checks which passages the user can access and supplies the selected text to the model. The model then generates an answer using that context. We can inspect the search results and the answer separately to find out where a mistake happened.
The original RAG research used a specific learned retrieval-and-generation setup. In application development, the term is also used more broadly for pipelines that retrieve documents and place them in a prompt. Distinguish the paper's method from the general engineering pattern.
Answer a question from a policy document
The user asks, “Can I cancel on the day of the workshop?” The collection contains an old policy, a current policy, and a general welcome page. A useful pipeline must find the relevant cancellation passage, identify the current version, preserve the deadline and exceptions, and connect the answer to that evidence.
If the current policy says “Cancel at least 24 hours before the start, except when the organizer cancels,” an answer of “No cancellations are allowed” is too broad. The correct passage reached the model, but the interpretation failed. Improving search alone would not repair that error.
Finding a passage and interpreting it
Retrieval-augmented generation is a chain of decisions. Documents are prepared, candidates are retrieved, selected evidence enters the request, and a model writes an answer. A failure at any stage can appear to the user as the same wrong sentence.
Inspect the earliest boundary where the necessary information is missing or misused. If the correct passage reached the model, changing the vector store may not address the observed failure. The experiment separates retrieving one relevant result from retrieving all needed support, and the challenge follows the answer through the remaining boundaries.
The retrieved passage contains the correct room and time.
Inspect whether both details survive context construction and answer generation.
You can distinguish a retrieval miss from lost context or unsupported generation.
From a policy document to an answer
Suppose a member asks, 'Can I get a refund for tomorrow's class?' The policy says refunds are possible within 14 days of purchase, but only before the class begins. A complete answer needs the policy, the purchase date and the class start status. Retrieving a paragraph that mentions refunds does not supply all three. Write the needed facts before you inspect the result list.
Now separate three failures. In the first, the policy document never enters the index because extraction failed. In the second, it is indexed correctly but the retriever returns another policy. In the third, the correct passage reaches the model but the answer ignores the class-start condition. All three can produce the same wrong sentence, yet they require different repairs. The source trace is what distinguishes them.
A citation is useful when it lets someone repeat this check. The cited passage must support the actual claim, and its revision must be the one used for the answer. A real citation beside an unsupported sentence is still a failure. Check important claims individually instead of treating the presence of any source link as proof that the whole response is grounded.
For the first working version, show the retrieved passages even if you have not added generation yet. Ask whether a person can answer from them. That gives you a direct test of the evidence layer. Once that layer works, add answer generation and evaluate whether it preserves the conditions, identifies missing facts and cites the support rather than filling gaps with plausible policy language.
Evaluate retrieval, answers, and citations
Retrieval quality asks whether the needed evidence appears in the selected set. Answer quality asks whether the response answers the question correctly from the available evidence. Citation quality asks whether the cited passages support the attached claims.
A fluent answer with an unrelated citation fails the third test. A missing-answer response when the source collection truly lacks the fact can be correct behavior. A correct answer produced from unsupported model memory may still violate a product promise to answer only from approved documents.
Keep these outcomes separate when diagnosing a system. Otherwise a single score can hide whether the problem is indexing, ranking, source selection, or generation.
Prepare documents for search
Ingestion means reading source material, extracting usable text, splitting it when necessary, and storing searchable records with metadata. Preserve document identity, version, source location, and access scope. A passage detached from its title or exception may be difficult to interpret even if it is retrieved.
Permissions must be applied before unauthorized content reaches the model. A prompt asking the model to ignore another user's file is not a substitute for retrieval authorization. Updates and deletions also need to propagate into the index so old or revoked content is not served indefinitely.
Choose a retrieval method
Lexical search is useful for exact terms, identifiers, and distinctive phrases. Dense embedding search can connect semantically related wording. Hybrid retrieval combines signals. Reranking can inspect a smaller candidate set more closely.
No method guarantees relevant evidence. A query about “money back” may need semantic matching to a “refund” policy; a query about policy ID RF-204 benefits from exact matching. Build tests that represent both kinds of request.
When to use RAG or fine-tuning
RAG changes the evidence supplied at inference time. Fine-tuning changes model parameters. They can be combined: a tuned model may learn a response style or extraction task while retrieval provides current private facts.
Neither is a universal cure for hallucinations. A fine-tuned model can still lack a new policy, and a RAG system can still retrieve or interpret the wrong passage. Choose based on the failure you observe and the update behavior you need.
Find the cause of a wrong answer
The answer gives the old 48-hour cancellation deadline. The trace shows the current 24-hour policy was never in the retrieved set. Where would you investigate first?
Follow the missing evidence
Check ingestion, version metadata, filtering, and retrieval ranking. Verify that the current policy is indexed and eligible for this user. Only after the right passage reaches the model can you meaningfully test whether generation uses it correctly. A stronger answer model cannot reliably quote evidence that the application did not supply.
Practice with feedback
Change how many results you keep
There are six results. Two passages contain needed evidence: one at rank 1 and another at the rank you choose. The other four are distractors. Keep the first k results.
The second supporting passage is missing.
Precision is relevant results divided by results kept. Recall is relevant results kept divided by the two relevant passages. A multi-part answer needs both passages here, even when the first result is already relevant.
What this experiment assumes. A fixed, labelled retrieval exercise. These labels are not similarity scores, and the example does not model a live search engine. Notes and recorded results here last until you leave this page.
Separate the three things a RAG system can get wrong
Your document assistant gives a wrong answer. The retrieved passage was correct and did contain the answer.
Evaluate retrieval, grounding, and answer quality independently.
Check your understanding
Your task
Design the three-way evaluation for one question set.
These notes stay on this page. Download them before leaving.
What to include
- Each measure has a definition someone else could apply
- The grounding measure checks claims against retrieved text, not against truth in general
- Both cross-cases are concrete
- The prioritisation follows from the numbers, not from preference
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Question set: 60 questions over the notices, each labelled by hand with the notice id that contains the answer, including 12 questions no notice answers.
1. Retrieval measure: recall@5. Did the labelled notice appear in the top 5? For the 12 unanswerable questions, the correct behaviour is that nothing clears the score threshold. 2. Grounding measure: for every sentence in the answer that states a fact, is it supported by the retrieved passages? I check by hand on a sample of 20 and count unsupported sentences. 3. Answer measure: does the answer state the correct day, time and room, judged against my labels?
A case where 1 passes and 2 fails: the right notice is retrieved, and the answer says 'usually starts at 18:30' when the notice says 18:30 with no hedge, plus adds 'bring your own thread', which no notice mentions. Retrieval is perfect; the answer invented a detail. A case where 1 fails and 3 accidentally passes: the retriever returns the wrong notice, but the answer happens to be right because the model knows the class from an earlier example in the prompt. It is correct and unsupported, and it will be wrong the moment the timetable changes.
Which I act on first: recall@5 is 0.71 and grounding is 0.94, so retrieval is the bottleneck. Improving the prompt would move the smaller number. If recall were 0.95 and grounding 0.70, I would leave the retriever alone.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, we will inspect the storage and search layer and distinguish a vector index from the broader responsibilities of a database.
Sources
Retrieval-Augmented Generation describes the original research method. The Sentence Transformers retrieval guide explains a practical retrieve-and-rerank arrangement.