Build the retrieval path before adding a larger stack

Before installing a vector database, can you prove that your pipeline preserves the right passage and source ID? A small exact baseline gives you something to compare with when the system becomes more sophisticated.

Before you begin: Understand RAG stages and Python functions. No external service is needed for the local retrieval example.

Define the evidence record

Use records with stable IDs, text, version, and access scope. For this exercise, all documents are fictional and public. In a private application, the trusted server must select eligible records using the authenticated identity before ranking.

The retriever returns evidence, not a final answer. This boundary lets you test search independently and makes it easier to swap lexical ranking for embeddings without changing the entire application.

Run a lexical baseline

This complete Python 3 program ranks tiny notices by overlapping query terms. It is not BM25, dense retrieval, or a full language-model RAG application. Its deliberately simple scoring exposes the handoff and its limitations.

# Runnable: Python 3, standard library.
import re

DOCUMENTS = [
    {'id': 'N1', 'version': 1, 'text': 'Refund requests need 24 hours notice.'},
    {'id': 'N2', 'version': 1, 'text': 'The class meets in Room 4.'},
    {'id': 'N3', 'version': 1, 'text': 'Bring a notebook to the class.'},
]

def terms(text):
    return set(re.findall(r'[a-z0-9]+', text.lower()))

def retrieve(question, limit=2):
    if not question.strip() or limit <= 0:
        raise ValueError('Use a question and a positive result limit')
    query = terms(question)
    scored = [(len(query & terms(d['text'])), d) for d in DOCUMENTS]
    scored.sort(key=lambda item: (-item[0], item[1]['id']))
    return [dict(d) for score, d in scored[:limit] if score > 0]

def context(records):
    return '\n\n'.join(
        f"[{d['id']} v{d['version']}] {d['text']}" for d in records)

assert retrieve('refund notice')[0]['id'] == 'N1'
assert retrieve('astronomy') == []
print(context(retrieve('refund notice')))

The output includes the source ID and version alongside text. Empty retrieval remains empty; the program does not invent a fallback document to keep the interface populated.

Connect the answer model

Use the model-call pattern from the earlier LangChain application. Supply the retrieved context and question, require claims to cite the record IDs, and define missing-evidence behavior. If no evidence is retrieved, you can return a clear “no matching source” state before making a generation call.

A nonempty result is not necessarily enough evidence. The word “class” appears in multiple notices and can retrieve a topically related passage that does not answer the question. Evaluate whether the selected text supports the specific answer, not merely whether search returned something.

Find the baseline's weakness

Ask “Can I get my money back?” The simple term-overlap system may miss the refund notice because the wording differs. This gives you a concrete reason to test dense retrieval or query expansion.

Now ask “Where is class N2?” Exact identifiers may make lexical matching especially useful. A dense retriever should not replace exact matching blindly. Preserve the baseline and compare task categories when adding a more flexible method.

Keep ingestion separate from querying

For larger sources, ingestion extracts text and creates records before users ask questions. Query-time retrieval should not repeatedly parse every file unless the collection is intentionally tiny. Preserve source offsets or other locators so a citation can open the relevant passage.

When a notice changes, update its version and remove or mark superseded records according to your policy. Test that queries do not keep returning stale passages from an old index. A retrieval pipeline needs an update path, not only an initial import script.

Complete the experiment

Write five questions: an exact match, a paraphrase, an exact ID, a missing answer, and an ambiguous question. For each, record the source that should be returned or why no answer is available. Run the retriever and inspect misses before adding the generator.

Next, you will turn this pipeline into a document-question project with a complete evidence and evaluation record.

References

The RAG paper motivates retrieval-conditioned generation. The LangChain model guide supplies the current invocation interface for the optional answer-model stage.

Continue to the next lesson.