Reading InstructGPT: helpfulness is measured, not assumed

What does it mean when reviewers prefer a smaller post-trained model to a larger base model? The InstructGPT paper helps answer that question through a specific prompt distribution and review process. Its result is useful precisely when we preserve those conditions.

Before you begin: You understand supervised instruction tuning, reward models, and RLHF.

The paper was published in 2022. Its experiments help explain the rise of instruction-following assistants. They are historical evidence under a particular task distribution, model family, and annotation process, not a leaderboard for current products.

Follow the three stages

First, human demonstrations provide examples for supervised fine-tuning. Second, reviewers compare model outputs, and those comparisons train a reward model. Third, a policy is optimized against that reward using reinforcement learning, with reference-related constraints and additional training choices described in the paper.

The authors also study a variant that mixes a pretraining objective into policy optimization. This addresses a practical tension: improving preferred assistant behavior can reduce performance on other tasks. Post-training is a tradeoff to evaluate, not a free improvement on every dimension.

Do not blur these stages into “humans tell the model the answer.” Demonstrations and comparisons provide different signals. The reward model then generalizes beyond the exact comparisons it saw, which introduces another source of error.

Ask whose preferences were measured

Read how prompts were collected and how labelers were selected and instructed. A model trained on those preferences may not represent every user's values, language, or expectations. Reviewer agreement and rubric design are part of the experiment.

A preference result means reviewers favored one output under the evaluation procedure. It does not automatically mean every factual statement was verified. Read the paper's separate analyses of truthfulness, toxicity, and other behavior instead of collapsing them into one word such as “aligned.”

Inspect a paired comparison

Imagine forty fictional comparisons: model A wins twenty-four, B wins twelve, and four are ties. You need to state how ties enter the reported rate.

# Runnable: Python 3, standard library.
wins, losses, ties = 24, 12, 4
total = wins + losses + ties
half_credit = (wins + 0.5 * ties) / total
decisive_only = wins / (wins + losses)
print(round(half_credit, 3), round(decisive_only, 3))
assert half_credit == 0.65

Both calculations can be described honestly, but they answer slightly different questions. These are invented teaching counts, not InstructGPT results. For a real paper table, preserve its exact definition and report uncertainty and sample size.

Change the tie rule, keep the judgments

The fictional comparison gives A 24 wins, 12 losses, and four ties. Counting ties as half a win yields 65 percent. Excluding ties yields about 66.7 percent. Counting only outright wins over all comparisons yields 60 percent. No reviewer changed a decision between these calculations.

Next, ask whether the compared answers were checked for factual support independently of preference. A response can win on clarity while containing a subtle error. This distinction is why the paper's separate analyses deserve separate notes in your reading summary.

Carry the method into an application

For our workshop assistant, use preferences to assess clarity and helpful handling of uncertainty, while separately checking factual support and tool outcomes. If reviewers prefer “Your booking is confirmed” after a failed tool call, the rubric needs correction. Pleasantness cannot substitute for an actual booking result.

Keep application evaluations outside the training loop. Include prompts from users unlike the people who wrote the demonstrations. Test harmless requests that might be over-refused as well as requests requiring limits.

Exercise: a smaller post-trained model is preferred to a larger base model on assistant prompts. Does that prove the smaller model is better at every task?

Next, build an adaptation project with this separation between training signals and product outcomes built in from the start.

Sources

Read Training Language Models to Follow Instructions with Human Feedback, including its methods and limitations. Use the paper's actual tables for reported results rather than approximate reconstructions.

Continue: Project: Fine-Tune Your Own Model.