Choose an encoder output that matches the task

You load an encoder and get one vector per token. Your search index needs one vector per document. Which vector should you store? Taking the first one because it is convenient may produce a system that runs but retrieves poorly.

Before you begin: Understand BERT and contextual representations.

Begin with the output contract

An encoder turns an input sequence into contextual token representations. A downstream component decides how those representations become a task result. For classification, a head produces label scores. For token labeling, a head predicts categories at positions. For retrieval, a trained pooling and embedding setup produces a sequence representation.

The architecture alone does not determine whether cosine similarity between two outputs is meaningful for your task. The training objective, pooling, normalization, tokenization, and maximum input length all matter.

Pool with the mask in mind

Pooling combines token vectors into a sequence vector. Mean pooling averages valid token representations. Padding tokens should not contribute just because they occupy array positions. Other models use a special token, weighted pooling, or a different learned representation.

This complete Python 3 example shows masked mean pooling with invented two-dimensional token states. It is not a sentence-embedding model.

# Runnable: Python 3, standard library.
def mean_pool(states, mask):
    if len(states) != len(mask) or not states:
        raise ValueError('States and mask must align')
    selected = [s for s, keep in zip(states, mask) if keep]
    if not selected:
        raise ValueError('At least one token must be valid')
    width = len(selected[0])
    if not width or any(len(s) != width for s in states):
        raise ValueError('State widths must match')
    return [sum(s[j] for s in selected)/len(selected)
            for j in range(width)]

states = [[2, 0], [0, 2], [100, 100]]
assert mean_pool(states, [True, True, False]) == [1, 1]
print(mean_pool(states, [True, True, True]))

The last state represents padding in the intended setup. Including it distorts the result to [34, 34]. Real padding states need not be this dramatic; the example makes the error visible.

Choose how a query meets a document

A bi-encoder encodes queries and documents separately into vectors. Document vectors can be precomputed, making large-scale retrieval practical. The similarity function then scores pairs without jointly reading all their tokens at query time.

A cross-encoder reads a query-document pair together and produces a relevance score or label. It can model detailed interactions between the two texts, but typically requires a separate model pass for each candidate pair. That often makes it useful for reranking a smaller retrieved set.

These are computational arrangements, not promises that one model always wins. Training and evaluation determine the quality-cost tradeoff in your setting.

Inspect truncation as part of the task

An encoder with a finite input limit cannot represent text it never receives. If a document's crucial exception occurs beyond truncation, changing the similarity metric will not recover it. Split or select text deliberately and record how evidence maps back to the original document.

For pairwise scoring, the query and document may share a combined token budget. A long query can reduce how much document text fits. Check the tokenizer's pair-truncation behavior instead of assuming both full texts are present.

Build a small retrieval check

Use three notices: “refunds allowed,” “refunds not allowed,” and “room location.” Query for the refund policy. A useful retriever should find the policy notices, but similarity alone may not distinguish which one is authoritative or current. Reranking and source metadata help with relevance; answer generation must still interpret the actual rule.

Choose the missing component

Your raw BERT vectors give poor semantic search, but a model trained for sentence similarity performs better. Does this prove BERT's encoder architecture cannot support retrieval?

Next, we will move from model components to application frameworks and ask what a framework simplifies without hiding the underlying request.

Sources

Sentence-BERT studies sentence representations for similarity. The Sentence Transformers retrieve-and-rerank guide explains bi-encoder and cross-encoder roles.

Continue to the next lesson.