Reinforcement learning from human feedback

Two answers are equally fluent, but one preserves uncertainty and the other invents a detail. How can preference feedback teach the distinction, and where can that signal go wrong? Follow the label, the learned reward, and the policy as separate stages.

Before you begin: You understand supervised fine-tuning, probability, and held-out evaluation.

Reinforcement learning from human feedback, or RLHF, is a family of approaches. A common language-model pipeline begins with supervised demonstrations, trains a reward model from comparisons, and optimizes the assistant against that learned reward. The details vary across systems.

Collect preferences with an explicit reason

Show reviewers responses to the same input and ask them to compare under a stated rubric. Separate factual support, instruction following, clarity, and appropriate handling of uncertainty. Otherwise “preferred” may mostly mean longer, friendlier, or more confident.

Allow for ties and disagreement in the data process. People can reasonably disagree about style, and they can also miss factual mistakes. A preference label is evidence about a judgment under particular conditions, not an objective certificate that an answer is correct.

Which part of a reward score is identified by a comparison?

A reward model learns to predict those judgments

A common pairwise model assigns a scalar score to each response and uses the score difference to predict preference. Under a logistic comparison model, a difference of zero gives probability one-half; a larger positive difference favors the chosen response.

# Runnable: Python 3, standard library.
import math

def preference_probability(chosen_score, rejected_score):
    difference = chosen_score - rejected_score
    return 1 / (1 + math.exp(-difference))

assert preference_probability(2.0, 2.0) == 0.5
print(round(preference_probability(3.0, 1.0), 3))

The displayed probability is a model of preference, not a probability that the chosen answer is true. A reward model can learn systematic reviewer mistakes as well as useful judgments.

Add the same constant to both rewards

The example compares scores 3 and 1. Replace them with 103 and 101. Predict the preference probability before running it: it stays the same because the difference is still two.

Now compare two incorrect answers. One may be preferred while both fail the task. Include absolute task checks alongside pairwise wins, and inspect high-reward failures. This small experiment explains why “reward increased” and “the assistant became more reliable” must remain separate claims.

Optimize without trusting the reward blindly

The assistant, often called the policy, generates responses and receives reward estimates. A reinforcement-learning optimizer adjusts it to improve the objective. Classical pipelines have used PPO and a penalty or constraint related to divergence from a reference policy.

The reference discourages large changes that exploit the reward model while damaging language behavior. It does not eliminate reward hacking. If the reward favors long confident answers, optimization may produce more long confident answers even when they are less factual.

Optimization also changes the distribution of responses. A reward model trained on one distribution may be unreliable on unusual responses produced by an aggressively optimized policy. Reassess the reward model and the policy on fresh examples.

Watch the disagreement between scores and outcomes

Track human-reviewed task success, factual support, appropriate refusal, and regressions independently of reward. Inspect cases with very high reward, not only low-scoring failures. They may reveal shortcuts the optimizer discovered.

Compare output lengths and phrasing. A rising reward accompanied by growing verbosity can be useful or suspicious depending on the task. Do not let the metric decide that distinction by itself.

Exercise: the policy's average reward rises, but reviewers find more unsupported claims. Where could the problem be?

Practice with feedback

import numpy as np

sig = lambda x: 1 / (1 + np.exp(-x))
bt_loss = lambda w, l: -np.log(sig(w - l))

print(bt_loss(2.4, 1.1))            # 0.2603
print(bt_loss(12.4, 11.1))          # 0.2603 - identical
print(bt_loss(2.4 * 3, 1.1 * 3))    # 0.0198 - scaling is NOT invariant

# Shifting both rewards changes nothing; scaling both does change the loss,
# which is why reward-model calibration matters even though its zero point
# does not. Practical consequence: never report 'mean reward went from 1.2 to
# 4.8' as evidence of improvement. It is a number on an arbitrary axis.

# Two signals I watch during optimisation:
#   1. Mean reward from the reward model (goes up by construction).
#   2. An independent held-out measure the reward model never saw: for us,
#      the evidence-following rate on 50 counterfactual pairs, scored by a
#      script, plus a weekly sample of 30 responses read by a person.
#
# The reward-hacking signature is divergence: reward climbing steadily while
# signal 2 is flat or falling. In our run, reward rose 40% over 800 steps
# while evidence-following fell from 0.88 to 0.71. Reading the samples showed
# why: responses had grown longer and more emphatic, and raters had preferred
# that, so the reward model had learned length and confidence as proxies for
# quality.
#
# Action on divergence: stop, take the last checkpoint where signal 2 was
# still healthy, and fix the preference data rather than the KL coefficient.
# Raising KL only slows the drift toward the same target.

Next, study DPO, which learns from preference pairs without a separate online reward-model optimization loop in its standard form.

Sources

InstructGPT documents a prominent RLHF pipeline. Deep Reinforcement Learning from Human Preferences develops the broader comparison-based approach. Their results do not imply that learned reward is a ground-truth measure.