Two answers are equally fluent, but one preserves uncertainty and the other invents a detail. How can preference feedback teach the distinction, and where can that signal go wrong? Follow the label, the learned reward, and the policy as separate stages.
Before you begin: You understand supervised fine-tuning, probability, and held-out evaluation.
Reinforcement learning from human feedback, or RLHF, is a family of approaches. A common language-model pipeline begins with supervised demonstrations, trains a reward model from comparisons, and optimizes the assistant against that learned reward. The details vary across systems.
Collect preferences with an explicit reason
Show reviewers responses to the same input and ask them to compare under a stated rubric. Separate factual support, instruction following, clarity, and appropriate handling of uncertainty. Otherwise “preferred” may mostly mean longer, friendlier, or more confident.
Allow for ties and disagreement in the data process. People can reasonably disagree about style, and they can also miss factual mistakes. A preference label is evidence about a judgment under particular conditions, not an objective certificate that an answer is correct.
A reward model learns to predict those judgments
A common pairwise model assigns a scalar score to each response and uses the score difference to predict preference. Under a logistic comparison model, a difference of zero gives probability one-half; a larger positive difference favors the chosen response.
# Runnable: Python 3, standard library.
import math
def preference_probability(chosen_score, rejected_score):
difference = chosen_score - rejected_score
return 1 / (1 + math.exp(-difference))
assert preference_probability(2.0, 2.0) == 0.5
print(round(preference_probability(3.0, 1.0), 3))
The displayed probability is a model of preference, not a probability that the chosen answer is true. A reward model can learn systematic reviewer mistakes as well as useful judgments.
Add the same constant to both rewards
The example compares scores 3 and 1. Replace them with 103 and 101. Predict the preference probability before running it: it stays the same because the difference is still two.
Why does the absolute score not certify quality?
In this pairwise logistic model, the probability depends on a score difference. Adding a shared constant does not change the comparison. A large raw number therefore is not an independently meaningful quality scale or probability of truth. Calibration and evaluation require additional evidence.
Now compare two incorrect answers. One may be preferred while both fail the task. Include absolute task checks alongside pairwise wins, and inspect high-reward failures. This small experiment explains why “reward increased” and “the assistant became more reliable” must remain separate claims.
Optimize without trusting the reward blindly
The assistant, often called the policy, generates responses and receives reward estimates. A reinforcement-learning optimizer adjusts it to improve the objective. Classical pipelines have used PPO and a penalty or constraint related to divergence from a reference policy.
The reference discourages large changes that exploit the reward model while damaging language behavior. It does not eliminate reward hacking. If the reward favors long confident answers, optimization may produce more long confident answers even when they are less factual.
Optimization also changes the distribution of responses. A reward model trained on one distribution may be unreliable on unusual responses produced by an aggressively optimized policy. Reassess the reward model and the policy on fresh examples.
Watch the disagreement between scores and outcomes
Track human-reviewed task success, factual support, appropriate refusal, and regressions independently of reward. Inspect cases with very high reward, not only low-scoring failures. They may reveal shortcuts the optimizer discovered.
Compare output lengths and phrasing. A rising reward accompanied by growing verbosity can be useful or suspicious depending on the task. Do not let the metric decide that distinction by itself.
Exercise: the policy's average reward rises, but reviewers find more unsupported claims. Where could the problem be?
Compare your reasoning
The preference rubric or labels may reward confidence, the reward model may generalize poorly, or optimization may exploit a proxy. Review all three stages. Training longer against the same flawed reward can make the problem worse.
Next, study DPO, which learns from preference pairs without a separate online reward-model optimization loop in its standard form.
Sources
InstructGPT documents a prominent RLHF pipeline. Deep Reinforcement Learning from Human Preferences develops the broader comparison-based approach. Their results do not imply that learned reward is a ground-truth measure.
Continue: DPO (Direct Preference Optimization).