DPO: learning which response to prefer

Can a preference objective improve while both candidate responses become less likely? DPO compares relative changes against a reference model. Working through that comparison prevents a lower loss from being mistaken for a complete measure of answer quality.

Before you begin: You understand log probabilities, supervised training, and the RLHF pipeline.

Direct Preference Optimization, or DPO, provides one way to do that. In its standard form, it uses a trainable policy, a frozen reference policy, and preference pairs. It adjusts how strongly the policy favors the chosen response relative to the rejected response, compared with the reference.

Begin with a valid preference pair

Both responses need the same prompt and available context. A chosen answer with extra hidden evidence is not a fair behavioral comparison. State why the response is preferred: factual support, task completion, or a clearly defined style criterion.

Evidence: Pottery is Saturday. Its time is unannounced.
Question: When should I arrive for pottery?
Chosen: It is on Saturday, but the arrival time is not given.
Rejected: Arrive on Saturday at 2 pm.

The important difference is the invented time. If all rejected answers are also shorter, badly spelled, and rude, the model may learn easier superficial clues instead. Include difficult pairs that isolate the desired distinction.

Read the objective in parts

For each response, sum the policy's log probabilities over the response tokens, conditioned on the prompt. Do the same with the frozen reference. Subtract the reference log probability from the policy log probability for each response, then compare chosen with rejected.

chosen_ratio = policy_logp(chosen) - reference_logp(chosen)
rejected_ratio = policy_logp(rejected) - reference_logp(rejected)
margin = beta * (chosen_ratio - rejected_ratio)
loss = -log(sigmoid(margin))

The prompt supplies context but should not accidentally be counted as response supervision. Padding, truncation, and tokenization must be handled consistently. The original objective uses sequence log probabilities; silently replacing sums with length averages changes the objective.

Work through one pair

# Runnable: Python 3, standard library.
import math

policy_chosen, policy_rejected = -8.0, -12.0
reference_chosen, reference_rejected = -10.0, -11.0
beta = 0.1
margin = beta * (
    (policy_chosen - reference_chosen)
    - (policy_rejected - reference_rejected)
)
loss = max(-margin, 0) + math.log1p(math.exp(-abs(margin)))
assert math.isclose(margin, 0.3)
assert loss < math.log(2)
print(round(loss, 4))

The policy improved the chosen response's relative log probability by two and reduced the rejected response's by one. The scaled margin is positive, so this pair's loss is below the zero-margin value of about 0.693. This is a local calculation, not a trained model's evaluation result.

Keep the margin, lower both probabilities

Change the policy log probabilities to -12 for the chosen response and -16 for the rejected response. Keep the reference at -10 and -11. The chosen ratio is now -2 and the rejected ratio -5. Their difference is still three, so beta 0.1 still produces margin 0.3.

Run the changed numbers through the local program and compare the loss with the original case. Then sketch a preferred answer that is merely less wrong than its alternative. Relative preference learning still needs data review and absolute checks on the behavior you want.

What beta controls

Beta appears in the preference objective and relates to the strength of reference regularization in the derivation. It also changes the loss's scale and gradient behavior during optimization. Avoid describing it as a universal knob where a larger value always produces either more or less behavioral change.

Its practical effect interacts with data, the reference, learning rate, and training duration. Tune it using held-out task outcomes and regressions, not only the preference loss. Changing beta to improve a chart is not evidence of a better assistant.

A better margin can still hide a problem

The objective is relative. A margin can improve by reducing the rejected response's probability even if the chosen response's probability also falls. Track both, and evaluate generated answers rather than only the training pairs.

Longer responses have more token log-probability terms. Length patterns in preference data can therefore matter. Do not assume the method is automatically free of verbosity bias. Inspect lengths and factual support together.

Standard offline DPO trains on an existing dataset. If the improved model generates different kinds of mistakes, the old pairs may no longer represent the hard cases. Collecting and reviewing fresh comparisons can help, but that is an additional data loop with its own evaluation needs.

Test the behavior you meant to teach

Exercise: DPO improves preference accuracy on held-out pairs, yet the assistant still invents times on new questions. What should you do next?

Keep the reference revision, dataset, response-token masking, beta, and optimizer settings with the adapter or model artifact. Next, read the earlier InstructGPT work to understand how demonstrations and preferences were evaluated together.

Sources

Direct Preference Optimization derives the standard objective. The TRL DPO Trainer documentation describes current implementation options; variants can use different losses and should be named explicitly.

Continue: Paper: Training Language Models to Follow Instructions.