BERT and Bidirectional Models
BERT changed NLP by showing that pretraining on large text corpora could create strong language understanding models.
Unlike left-to-right generation models, BERT reads both sides of a word.
Masked language modeling
BERT trains by hiding tokens and predicting them:
The user clicked the [MASK] button.
Because the model sees both left and right context, it builds strong representations for understanding.
BERT vs GPT-style models
| BERT-style | GPT-style |
|---|---|
| bidirectional encoder | causal decoder |
| great for understanding | great for generation |
| predicts masked tokens | predicts next tokens |
| used for embeddings/classifiers | used for chat/completion |
Why BERT still matters
Even in the LLM era, BERT-style models are useful for:
- classification
- search embeddings
- reranking
- moderation
- entity extraction
- smaller domain-specific systems
They are often faster and cheaper than chat models for narrow understanding tasks.
RoBERTa lesson
RoBERTa showed that training recipe matters:
- more data
- longer training
- better batches
- remove weak objectives
Sometimes "better training" beats "new architecture."
Knowledge check
Q1: What is masked language modeling?
Training the model to predict hidden tokens from surrounding context.
Q2: Why use BERT-style models today?
They are efficient for understanding tasks like classification, embeddings, and reranking.