A search interface returns five passages, but you cannot tell why they were retrieved or whether the first one contains the answer. Build a search-only project before adding generation. It makes ranking failures visible instead of letting a fluent answer hide them.
Before you begin: Understand embeddings, similarity metrics, and retrieval evaluation.
Prepare a small collection
Create a dozen short fictional notices with stable IDs and titles. Include exact identifiers, paraphrases, related non-answers, and a revised policy. Write at least six questions and label the passages that would help answer each one.
Keep these relevance judgments separate from any examples used to tune the system. The aim is to compare methods on new wording, not to memorize your own queries.
Can you explain why this passage was returned?
Search is easier to debug before generation hides the retrieved evidence inside a paragraph. Show the matching passages, their source locations and the query that produced them. Ask whether a person can answer the question from those results alone.
Use the retrieval controls to explore the difference between a concise list and complete support. Then build the challenge’s search interface around inspectable evidence. Empty and irrelevant result sets should be visible states, so the user can refine the query without mistaking a guessed answer for a successful search.
A search result shows only a document title and a high score.
Display the retrieved passage and a useful source location.
The user can judge whether the result contains the needed evidence.
Try a real embedding model
In an isolated Python environment, install sentence-transformers. The following complete integration example uses a small public model documented by the library. It downloads model files on first use and needs the package, network access, and sufficient local memory. Its API was checked against current documentation; model inference was not executed during this lesson's initial drafting.
python -m pip install sentence-transformers
from sentence_transformers import SentenceTransformer
documents = [
('N1', 'Refund requests need at least 24 hours notice.'),
('N2', 'The drawing class meets in Room 4.'),
('N3', 'Contact the organizer with general questions.'),
]
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
doc_vectors = model.encode([text for _, text in documents], normalize_embeddings=True)
query = model.encode(['How can I get my money back?'], normalize_embeddings=True)
scores = model.similarity(query, doc_vectors)[0].tolist()
ranking = sorted(range(len(documents)), key=lambda i: (-scores[i], documents[i][0]))
for i in ranking:
print(documents[i][0], round(scores[i], 4), documents[i][1])
No expected numerical scores are invented here. Inspect whether N1 ranks above the generic contact notice in your run. Record the installed versions and model revision used if you need reproducible comparison. This small sentence model is an educational candidate, not a universal recommendation for long-document retrieval.
Compare against the lexical baseline
Run the same queries through the preceding term-overlap retriever. Include an exact ID query and a paraphrase. A semantic model may improve one category while a lexical method remains useful for another.
Then test hybrid fusion if the error patterns justify it. Keep the methods independently runnable so you can explain what each contributes. Do not add a reranker only because the final architecture diagram looks more complete with one.
Measure ranking usefully
For a question with one relevant passage, record whether it appears in the top result and top three. For multiple relevant passages, compute recall at the chosen cutoff. Inspect missing-answer cases separately; returning some nearest neighbor does not establish that an answer exists.
Keep a small error table with the query, expected source IDs, retrieved IDs, and likely cause. Causes may include truncation, missing synonyms, ambiguous labels, stale records, or incomplete ingestion.
Present the evidence directly
Display title, short passage, source locator, and date or version when relevant. Label a similarity number as a retrieval score if you show it; do not turn 0.8 into “80% correct.” A user should be able to open the source and decide whether it answers the question.
On a phone, prioritize the query and readable result excerpts. Keep filters compact and make empty and failed-search states distinct. The search product should remain useful without an ornamental generated summary.
Complete the project
Deliver the collection, query judgments, reproducible search command, dependency record, and observed comparison. Include one failure you can explain and one change you tested because of that failure.
What if the dense model loses on exact IDs?
Keep exact matching or lexical retrieval for that requirement and test a hybrid route. A model is not defective merely because a different representation fits identifiers better. The project succeeds when you can justify the search design from the task's evidence.
Practice with feedback
Change how many results you keep
There are six results. Two passages contain needed evidence: one at rank 1 and another at the rank you choose. The other four are distractors. Keep the first k results.
The second supporting passage is missing.
Precision is relevant results divided by results kept. Recall is relevant results kept divided by the two relevant passages. A multi-part answer needs both passages here, even when the first result is already relevant.
What this experiment assumes. A fixed, labelled retrieval exercise. These labels are not similarity scores, and the example does not model a live search engine. Notes and recorded results here last until you leave this page.
Build a semantic search engine
Before building a question-answering layer, you build search that shows the matching passages and nothing else.
Ship ranked retrieval with a measured comparison against a lexical baseline.
Check your understanding
Your task
Write the evaluation and the result design for your search before adding any answering.
These notes stay on this page. Download them before leaving.
What to include
- The labelled set includes questions with no correct answer
- At least two metrics, one of them rank-sensitive
- All three retrieval approaches are measured on the same questions
- The gate for adding generation is a number, not a feeling
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Collection: 30 workshop notices plus 12 committee minutes, structure-aware chunked to at most 800 characters with heading paths, 260 chunks. Labelled questions: 60, labelled by me before any run, each tagged with the chunk id that fully answers it. 12 of the 60 have no answer in the collection.
Metrics: recall@3 and mean reciprocal rank, plus the false-hit rate on the 12 unanswerable questions (how often something clears the threshold anyway). Baseline lexical: recall@3 0.73, MRR 0.61, false hits 7/12. Semantic: recall@3 0.85, MRR 0.74, false hits 9/12. Better ranking, worse at knowing when to return nothing, because cosine always returns something. Hybrid (RRF): recall@3 0.91, MRR 0.80, false hits 6/12 with a minimum-lexical-overlap rule.
Result item design: the matched passage with query terms highlighted, the heading path above it, source title, version date, and a link that opens the document at that offset. No score shown to users; the score is in the debug view. What I would need before adding a generated answer: recall@3 above 0.90 (met) and the false-hit rate down to about 2 in 12. That second number is the one holding me back, and it is the right one to hold me back: a generation layer on top of a retriever that returns confident junk for unanswerable questions produces confident junk in prose.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, the agent-framework module will use retrieval as one bounded capability inside a larger control loop.
References
The Sentence Transformers similarity guide documents encoding and pairwise scores. Its retrieve-and-rerank guide describes a later extension.