Your organization changes its cancellation policy this morning. A language model's parameters do not automatically update. You can provide the new policy in the request, but what happens when there are thousands of documents and only a few are relevant?
Before you begin: Understand embeddings, model context, and source-grounded answering.
Retrieve before generating
Retrieval-augmented generation, or RAG, combines retrieval from an external collection with a generative model. At question time, a retriever selects candidate passages. The application supplies allowed evidence to the model, which produces an answer conditioned on that context.
The original RAG research used a specific learned retrieval-and-generation setup. In application development, the term is also used more broadly for pipelines that retrieve documents and place them in a prompt. Distinguish the paper's method from the general engineering pattern.
Follow one question through the system
The user asks, “Can I cancel on the day of the workshop?” The collection contains an old policy, a current policy, and a general welcome page. A useful pipeline must find the relevant cancellation passage, identify the current version, preserve the deadline and exceptions, and connect the answer to that evidence.
If the current policy says “Cancel at least 24 hours before the start, except when the organizer cancels,” an answer of “No cancellations are allowed” is too broad. The correct passage reached the model, but the interpretation failed. Improving search alone would not repair that error.
Separate three evaluations
Retrieval quality asks whether the needed evidence appears in the selected set. Answer quality asks whether the response answers the question correctly from the available evidence. Citation quality asks whether the cited passages support the attached claims.
A fluent answer with an unrelated citation fails the third test. A missing-answer response when the source collection truly lacks the fact can be correct behavior. A correct answer produced from unsupported model memory may still violate a product promise to answer only from approved documents.
Keep these outcomes separate when diagnosing a system. Otherwise a single score can hide whether the problem is indexing, ranking, source selection, or generation.
Prepare sources before search
Ingestion means reading source material, extracting usable text, splitting it when necessary, and storing searchable records with metadata. Preserve document identity, version, source location, and access scope. A passage detached from its title or exception may be difficult to interpret even if it is retrieved.
Permissions must be applied before unauthorized content reaches the model. A prompt asking the model to ignore another user's file is not a substitute for retrieval authorization. Updates and deletions also need to propagate into the index so old or revoked content is not served indefinitely.
Choose retrieval for the question
Lexical search is useful for exact terms, identifiers, and distinctive phrases. Dense embedding search can connect semantically related wording. Hybrid retrieval combines signals. Reranking can inspect a smaller candidate set more closely.
No method guarantees relevant evidence. A query about “money back” may need semantic matching to a “refund” policy; a query about policy ID RF-204 benefits from exact matching. Build tests that represent both kinds of request.
Keep RAG and fine-tuning distinct
RAG changes the evidence supplied at inference time. Fine-tuning changes model parameters. They can be combined: a tuned model may learn a response style or extraction task while retrieval provides current private facts.
Neither is a universal cure for hallucinations. A fine-tuned model can still lack a new policy, and a RAG system can still retrieve or interpret the wrong passage. Choose based on the failure you observe and the update behavior you need.
Diagnose a failed answer
The answer gives the old 48-hour cancellation deadline. The trace shows the current 24-hour policy was never in the retrieved set. Where would you investigate first?
Follow the missing evidence
Check ingestion, version metadata, filtering, and retrieval ranking. Verify that the current policy is indexed and eligible for this user. Only after the right passage reaches the model can you meaningfully test whether generation uses it correctly. A stronger answer model cannot reliably quote evidence that the application did not supply.
Next, we will inspect the storage and search layer and distinguish a vector index from the broader responsibilities of a database.
Sources
Retrieval-Augmented Generation describes the original research method. The Sentence Transformers retrieval guide explains a practical retrieve-and-rerank arrangement.