Encoder-Only Models (BERT, RoBERTa)
Encoder-only models read the whole input at once. They are excellent when the goal is understanding, not generating long text.
Encoder vs decoder
| Model type | Sees | Best for |
|---|---|---|
| encoder-only | full input both directions | classification, embeddings, reranking |
| decoder-only | previous tokens only | generation and chat |
| encoder-decoder | source plus generated target | translation and summarization |
Why BERT mattered
BERT used masked language modeling:
text
The capital of France is [MASK].
The model learns to fill missing tokens using both left and right context.
Where encoder models still win
Use encoder-style models for:
- sentiment classification
- semantic similarity
- search embeddings
- reranking retrieved passages
- named entity recognition
- moderation classifiers
- compact domain models
Why not use a chat model for everything?
Chat models are flexible but can be slower and more expensive. If the task is a stable classification or scoring task, encoder models can be cheaper, faster, and easier to evaluate.
Knowledge check
Q1: What is the strength of encoder-only models?
They understand the full input bidirectionally.
Q2: Name one production use for encoder-only models.
Embeddings, reranking, classification, moderation, or entity extraction.