Two sentences can be close in meaning while only one answers a question. “Refunds are available within 24 hours” and “Contact us with refund questions” concern the same topic. A retriever needs to learn more than broad topical similarity if the user asks for the deadline.
Before you begin: Understand similarity metrics and encoder pooling.
Ask what the model was trained to bring together
An embedding model maps inputs to vectors. Its training objective shapes which relationships those vectors express. A model trained for sentence paraphrase similarity may behave differently from one trained to match short queries with answer passages.
In contrastive training, the model is encouraged to score suitable pairs above unsuitable pairs. A query and a relevant passage form a positive pair. Other passages act as negatives. The choice of negatives matters: an unrelated recipe is easy to separate from a refund question, while a near-matching but non-answering policy paragraph is harder.
Some negatives may actually be relevant but unlabeled. Treating them as definitely wrong can teach a misleading boundary. Data quality and task definition remain important even when the loss function is mathematically clear.
Distinguish symmetric and asymmetric tasks
Comparing two sentences for paraphrase is roughly symmetric. Matching “When can I cancel?” to a long policy paragraph is asymmetric: the query and document play different roles. Some models use query/document prefixes, separate encoders, or training procedures designed for this distinction.
Use the model's documented encoding procedure. Omitting a required prefix can degrade retrieval while still producing the expected vector shape. Shape validation catches some bugs, but it does not prove the representation is being used correctly.
Did the ranking improve, or only the score scale?
Two embedding models can assign different numerical scores to the same pairs. A higher average score does not establish that relevant documents moved above irrelevant ones. Evaluate the ranked results against labels or task outcomes instead of comparing score magnitudes across models.
Try stretching a vector in the geometry exercise to see why the comparison operation matters. Then inspect the upgrade in the challenge: keep questions and documents fixed, rebuild representations consistently and measure retrieval quality. Mixing vectors from incompatible models is a separate error from choosing a poor threshold.
Average pair similarity rises after changing the embedding model.
Check whether relevant passages move up the result lists on labelled queries.
The upgrade may improve ranking, worsen it or merely shift the score distribution.
Inspect a small neighborhood
Create a query and four passages: the exact answer, a paraphrase of the answer, a topically related non-answer, and an unrelated passage. Compute their scores with your chosen model and inspect the ranking.
Do not supply expected numerical scores from a tutorial unless they were actually measured under the documented setup. The useful observation is which candidate outranks which and whether that relationship serves the task.
Add a negated policy and an exact identifier. These cases often reveal a gap between semantic closeness and answer usefulness. A hybrid search or reranking stage may help, but test the specific failure before adding components.
Consider dimensions and storage
Vector width affects storage and computation, but more dimensions do not guarantee better retrieval. Some models support documented dimension reduction or truncation procedures; arbitrary truncation of an unrelated model is not necessarily valid.
For a rough raw-storage estimate, one million vectors with 384 float32 coordinates require about 1.536 billion bytes for coordinates alone. IDs, text, metadata, indexes, replicas, and runtime overhead add more. This arithmetic is not a database capacity promise.
Normalization and dtype also affect behavior. Quantized vector storage may save space while changing ranking accuracy. Compare against the original representation on a held-out query set.
Plan an embedding migration
When replacing a model, create a separate index with the new vectors and its configuration. Evaluate both indexes using the same questions and relevance judgments. Do not mix vectors from incompatible spaces in one collection just because their dimensions match.
Keep source records and stable IDs independent of vector implementation. That makes re-embedding, rollback, and comparison easier. Record the model revision, preprocessing, chunking, and metric with the index version.
Diagnose a misleading improvement
A new model ranks related documents more closely together in a two-dimensional visualization, but question-answer retrieval gets worse. Which evidence should guide the decision?
Prefer the task over the picture
Use retrieval outcomes on representative questions. A dimensionality-reduction plot can distort distances and is not a direct measure of answer relevance. Investigate the ranking failures, query/document formatting, and truncation behavior before concluding that the model is better or worse from the visualization alone.
Practice with feedback
Turn a vector without stretching it
The query is [1, 0]. The candidate is [length × cos(angle), length × sin(angle)]. Cosine divides the dot product by the lengths of both vectors.
Candidate vector: [0.707, 0.707]
At 90 degrees the dot product is zero. At 180 degrees it is negative. Stretch the candidate at a fixed angle: the raw dot product changes, but the cosine stays fixed.
What this experiment assumes. Two-dimensional geometry, not measured semantic similarity. A negative cosine is a direction relationship, not proof that two statements contradict each other. Notes and recorded results here last until you leave this page.
Diagnose a misleading embedding upgrade
You swap embedding models. Average similarity scores rise from 0.62 to 0.81 and the team declares victory.
Match the training objective to your task shape and detect a false improvement.
Check your understanding
Your task
Design the comparison that would have caught the false improvement.
These notes stay on this page. Download them before leaving.
What to include
- The metric is retrieval quality on labelled questions, not a similarity level
- The separation check compares two distributions, not one average
- The migration re-embeds everything and can run alongside the old index
- The rollback trigger is a measured threshold
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
What I measure instead: recall@5 and mean reciprocal rank on the 60 labelled questions. Similarity level does not appear anywhere in the decision.
Separation check: for 200 known-relevant pairs and 200 known-irrelevant pairs, print both distributions. Old model: relevant mean 0.62 (sd 0.09), irrelevant mean 0.28 (sd 0.11). Gap 0.34. New model: relevant mean 0.81 (sd 0.05), irrelevant mean 0.72 (sd 0.07). Gap 0.09. The new model scores everything higher and separates almost nothing. That is what the celebrated 0.81 actually was.
Task shape: asymmetric. Our inputs are 8-word questions against 200-word passages, and the winning candidate is a model with an explicit query instruction prefix. I confirmed by checking that swapping the prefix between query and passage measurably degrades recall, which only happens for a genuinely asymmetric model.
Migration plan: 1. Re-embed all 200,000 chunks into a second index tagged with the new model id. Nothing is deleted. 2. Run both indexes on the 60 labelled questions and on a week of live queries in shadow mode, comparing recall@5. 3. Switch reads only if the new index wins, then retire the old index after two weeks.
Rollback trigger: recall@5 more than 2 points below the old index on the labelled set, or any single question class dropping more than 10 points. Both are checked automatically before the switch and daily for two weeks after.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, we will compare database options through a reproducible workload and an operational decision record.
Sources
Dense Passage Retrieval studies query-passage representation learning. Sentence-BERT studies sentence-level representations. The raw-storage calculation above is a transparent estimate, not a reported benchmark.