Build search you can inspect before adding answers

A search interface returns five passages, but you cannot tell why they were retrieved or whether the first one contains the answer. Build a search-only project before adding generation. It makes ranking failures visible instead of letting a fluent answer hide them.

Before you begin: Understand embeddings, similarity metrics, and retrieval evaluation.

Prepare a small collection

Create a dozen short fictional notices with stable IDs and titles. Include exact identifiers, paraphrases, related non-answers, and a revised policy. Write at least six questions and label the passages that would help answer each one.

Keep these relevance judgments separate from any examples used to tune the system. The aim is to compare methods on new wording, not to memorize your own queries.

Try a real embedding model

In an isolated Python environment, install sentence-transformers. The following complete integration example uses a small public model documented by the library. It downloads model files on first use and needs the package, network access, and sufficient local memory. Its API was checked against current documentation; model inference was not executed during this lesson's initial drafting.

python -m pip install sentence-transformers

from sentence_transformers import SentenceTransformer

documents = [
    ('N1', 'Refund requests need at least 24 hours notice.'),
    ('N2', 'The drawing class meets in Room 4.'),
    ('N3', 'Contact the organizer with general questions.'),
]
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
doc_vectors = model.encode([text for _, text in documents], normalize_embeddings=True)
query = model.encode(['How can I get my money back?'], normalize_embeddings=True)
scores = model.similarity(query, doc_vectors)[0].tolist()
ranking = sorted(range(len(documents)), key=lambda i: (-scores[i], documents[i][0]))
for i in ranking:
    print(documents[i][0], round(scores[i], 4), documents[i][1])

No expected numerical scores are invented here. Inspect whether N1 ranks above the generic contact notice in your run. Record the installed versions and model revision used if you need reproducible comparison. This small sentence model is an educational candidate, not a universal recommendation for long-document retrieval.

Compare against the lexical baseline

Run the same queries through the preceding term-overlap retriever. Include an exact ID query and a paraphrase. A semantic model may improve one category while a lexical method remains useful for another.

Then test hybrid fusion if the error patterns justify it. Keep the methods independently runnable so you can explain what each contributes. Do not add a reranker only because the final architecture diagram looks more complete with one.

Measure ranking usefully

For a question with one relevant passage, record whether it appears in the top result and top three. For multiple relevant passages, compute recall at the chosen cutoff. Inspect missing-answer cases separately; returning some nearest neighbor does not establish that an answer exists.

Keep a small error table with the query, expected source IDs, retrieved IDs, and likely cause. Causes may include truncation, missing synonyms, ambiguous labels, stale records, or incomplete ingestion.

Present the evidence directly

Display title, short passage, source locator, and date or version when relevant. Label a similarity number as a retrieval score if you show it; do not turn 0.8 into “80% correct.” A user should be able to open the source and decide whether it answers the question.

On a phone, prioritize the query and readable result excerpts. Keep filters compact and make empty and failed-search states distinct. The search product should remain useful without an ornamental generated summary.

Complete the project

Deliver the collection, query judgments, reproducible search command, dependency record, and observed comparison. Include one failure you can explain and one change you tested because of that failure.

Next, the agent-framework module will use retrieval as one bounded capability inside a larger control loop.

References

The Sentence Transformers similarity guide documents encoding and pairwise scores. Its retrieve-and-rerank guide describes a later extension.

Continue to the next lesson.