RLHF (Reinforcement Learning from Human Feedback)
RLHF is one way to make a base language model behave more like a helpful assistant.
The model learns not just to predict text, but to prefer responses humans rate better.
Pipeline
text
base model -> supervised fine-tuning -> preference data -> reward model -> policy optimization
Preference data
Humans compare answers:
text
prompt -> answer A vs answer B -> which is better?
The comparisons train a reward model.
Reward model
The reward model predicts which responses humans would prefer. Then policy optimization nudges the language model toward high-reward answers.
Failure modes
| Failure | Meaning |
|---|---|
| reward hacking | model exploits reward flaws |
| over-refusal | model refuses harmless requests |
| sycophancy | model agrees too much |
| verbosity bias | longer answers score better even when worse |
| preference drift | model optimizes judge quirks |
Why DPO became popular
DPO simplifies preference optimization by avoiding a separate reward-model training loop. RLHF still matters because it explains much of modern assistant behavior.
Knowledge check
Q1: What does the reward model learn?
It learns to predict which answers humans prefer.
Q2: What is reward hacking?
Optimizing the reward signal without actually improving the intended behavior.