DPO (Direct Preference Optimization)
DPO trains models from preference pairs without a separate reward model and RL loop.
Preference pair:
text
prompt -> chosen answer vs rejected answer
The model learns to increase the likelihood of the chosen answer relative to the rejected one.
Why DPO became popular
| RLHF | DPO |
|---|---|
| reward model + PPO loop | direct preference loss |
| more moving parts | simpler pipeline |
| harder to stabilize | easier to reproduce |
| still powerful | widely used for alignment |
Good preference data
Pairs should show meaningful differences:
- safer vs unsafe
- grounded vs hallucinated
- concise vs rambling
- correct tool use vs wrong tool use
- follows format vs ignores format
Limits
DPO is not magic. It still depends on data quality, evals, and careful rollout.
Knowledge check
Q1: What data does DPO use?
Chosen and rejected responses for the same prompt.
Q2: Why is DPO simpler than RLHF?
It skips separate reward-model training and RL policy optimization.