“The visitor left the bag beside the [MASK] before entering the classroom.” The words after the gap help constrain what belongs there. BERT's pretraining objective makes use of that two-sided context without letting the model simply read the selected target token unchanged in every case.
Before you begin: Understand bidirectional encoders and causal decoders.
Read both sides of the input
BERT is a transformer encoder trained to build bidirectional representations. Each valid input position can combine information from both earlier and later positions. This differs from a causal language model, where a position cannot read future target tokens.
Bidirectional access is appropriate when the complete input already exists, such as a sentence to classify or a document to annotate. It would leak information if used unmodified to score next-token predictions while the future target was visible.
Inspect the corruption objective
In the original BERT setup, a subset of input tokens is selected for prediction. Selected tokens are not all replaced in the same way: many become a mask token, some are replaced by random tokens, and some remain unchanged. Loss is calculated on the selected targets. This reduces the mismatch that would arise if pretraining always used a mask symbol absent from downstream inputs.
The original paper also used a next-sentence prediction objective. Later work such as RoBERTa changed the training recipe and removed that objective in its setup. This is a reminder to distinguish the original architecture from one particular recipe and from later variants.
Follow a contextual token
Compare “The bank approved the loan” with “The bank was muddy after the flood.” A static token embedding starts from the same token ID for “bank,” subject to tokenizer details. Encoder layers produce different contextual representations because the surrounding tokens differ.
Those representations can help a downstream classifier distinguish financial questions from river reports. The model does not need a separate hand-written rule for every use, but it still needs suitable training and evaluation for the task. Ambiguous or incomplete context can remain ambiguous.
Add the right task head
A sequence-classification model maps a sequence representation to label scores. A token-classification model maps positions to labels such as entity categories. An extractive question-answering setup predicts answer boundaries in a passage. These heads use the encoder in different ways.
For a subword tokenizer, a person's name may occupy several tokens. A token-labeling pipeline must decide how word-level labels align with those pieces and how predictions are combined afterward. Ignoring alignment can make a correct-looking tensor shape produce incorrect annotations.
Do not assume the raw representation at a special classification token is already an ideal sentence embedding for cosine search. Sentence-embedding models are usually trained or adapted for that objective, with appropriate pooling and normalization.
Work through a task choice
You need to label whether a short message reports a broken joining link. An encoder classifier can return fixed label scores without generating an explanation. A chat model can solve the task through prompting but has a different output and runtime path.
Compare both if the task warrants it. Use representative messages, label definitions, and latency measurements on the actual hardware or service. “Encoders are always cheaper” is too broad; sequence length, batching, model size, and deployment all matter.
Test the leakage distinction
A researcher trains an unmasked bidirectional encoder to predict every visible input token and reports excellent accuracy. Does that demonstrate useful masked-language learning?
Ask whether the target was hidden
No. If the target is plainly available at the position, the network can learn to copy it. A prediction task must control what information is visible and where the loss is applied. The original masking setup makes the problem meaningfully different from reconstructing an unchanged visible token at every position.
Next, we will examine reasoning-focused systems through their measurable behavior and inference budgets, rather than an assumed universal internal architecture.
Sources
BERT describes the original objectives and task adaptations. RoBERTa investigates the influence of training choices. Sentence-BERT addresses sentence representations for similarity.