How to measure embedding similarity

Two documents receive scores of 0.8 from different search systems. Are they equally relevant? Not necessarily. A score has meaning only in relation to the embedding model, metric, preprocessing, and task.

Before you begin: Know vectors and elementary arithmetic. Python is optional for the worked calculation.

Compare three measurements

The dot product multiplies corresponding vector coordinates and sums them. It is affected by direction and magnitude. Cosine similarity divides the dot product by the product of vector lengths, emphasizing direction. Euclidean distance measures straight-line distance; smaller values mean closer vectors.

For a query [1, 0], document A [2, 0] has dot product two and cosine one. Document B [1, 1] has dot product one and cosine about 0.707. Their Euclidean distances from the query are both one. The metrics can tie or rank candidates differently because they measure different properties.

What does a threshold actually decide?

Work with normalized vectors

A unit-normalized vector has length one. For two unit vectors, squared Euclidean distance equals 2 - 2 × dot_product. Their dot product also equals cosine similarity. Under those conditions, maximizing cosine and minimizing Euclidean distance produce equivalent rankings.

The conditions matter. If only documents are normalized or the model expects magnitude to carry information, the relationship changes. Follow the embedding model's guidance and the database metric configuration.

This complete Python 3 program checks the unit-vector relationship with a small example.

# Runnable: Python 3, standard library.
import math

def normalize(v):
    length = math.sqrt(sum(x*x for x in v))
    if length == 0:
        raise ValueError('Cannot normalize a zero vector')
    return [x/length for x in v]

a = normalize([1, 0])
b = normalize([1, 1])
dot = sum(x*y for x, y in zip(a, b))
distance_squared = sum((x-y)**2 for x, y in zip(a, b))
assert math.isclose(distance_squared, 2-2*dot)
print(round(dot, 3), round(distance_squared, 3))

Keep the embedding spaces compatible

Queries and documents must be encoded using a compatible model and procedure. Some retrieval models use different prefixes or encoding paths for queries and documents. Equal vector dimensions do not prove compatibility between unrelated checkpoints.

Record model revision, dimensions, normalization, tokenizer behavior, and preprocessing with the index. If any of these changes, evaluate whether re-embedding is required. A silent mixture can return plausible-looking numbers with poor retrieval.

Do not read probability into every score

A cosine score of 0.8 is not an 80% probability that the document answers the question. Similarity scores are not automatically calibrated confidence values. Thresholds must be evaluated for the chosen model, corpus, and query distribution.

Likewise, a reranker's output may be a raw logit or another score depending on the model and activation. Do not assume every relevance model returns a probability between zero and one. Check its documented output and calibration before displaying a confidence percentage.

Inspect the semantic boundary

“The class is cancelled” and “The class is not cancelled” are closely related in topic. A retriever may rank both highly. The answerer must interpret the negation and identify the authoritative notice. Similarity helps locate candidate evidence; it does not resolve truth or document authority by itself.

Include exact IDs, paraphrases, negation, and irrelevant but topically similar passages in your evaluation. A clean demonstration with obvious synonyms is not enough to choose a production threshold.

Choose a threshold responsibly

You see that every relevant example in a small development set scores above 0.75. Can you treat 0.75 as a universal threshold for all future collections?

Practice with feedback

import numpy as np

scores = np.array([cosine(embed(a), embed(b)) for a, b, _ in pairs])
labels = np.array([d for _, _, d in pairs])

for t in np.arange(0.60, 0.96, 0.05):
    pred = scores >= t
    tp = int((pred & (labels == 1)).sum())
    fp = int((pred & (labels == 0)).sum())
    fn = int((~pred & (labels == 1)).sum())
    prec = tp / max(tp + fp, 1)
    rec = tp / max(tp + fn, 1)
    print(f't={t:.2f} precision={prec:.2f} recall={rec:.2f} fp={fp} fn={fn}')

# t=0.75 precision=0.62 recall=0.95   <- 14 false merges
# t=0.85 precision=0.91 recall=0.83
# t=0.90 precision=1.00 recall=0.61
#
# Chosen: 0.90. A false merge hides a real notice from members and is found
# only when someone turns up to the wrong room. A missed duplicate leaves two
# notices on the page, which a volunteer notices and fixes in seconds. The
# expensive error is the false merge, so I take precision.
#
# Recorded with the threshold: embedding model 'bge-small-en-v1.5', normalised
# vectors, measured on 48 labelled pairs, 2026-09-10. Re-measure if the model
# changes: the number does not transfer.

Next, we will assemble a small retrieval pipeline with source identities before introducing a more complex database or embedding service.

Sources

Sentence-BERT studies sentence representations for similarity. The Faiss metric notes explain distance choices and the role of normalization.