Fine-tuning: teach a behavior you can measure

A clear prompt still fails on a repeated, well-defined task. Can reviewed demonstrations teach the missing behavior? Fine-tuning is an experiment about that behavior, so the first artifact should be a target you can inspect.

Before you begin: Complete transfer learning and understand training, validation, and test splits.

Fine-tuning continues training from an existing model. Full fine-tuning updates all or most model parameters; adapter methods update a smaller set. In both cases, the objective and examples determine what the model is encouraged to do.

Define one observable outcome

Avoid “make the model smarter.” A useful goal might be: given a request and a verified workshop record, produce the requested fields, preserve the record's values, and mark missing information as unknown.

Create a baseline evaluation before collecting more training examples. Include ordinary requests, paraphrases, missing fields, contradictory evidence, and out-of-scope questions. Decide how to measure correctness separately from formatting.

Make each training record self-contained

A chat example should include the instructions and context available at deployment. Here is a fictional record, shown as data rather than an API call.

{
  "messages": [
    {"role": "system", "content": "Use the supplied record. Do not invent a time."},
    {"role": "user", "content": "Record: pottery, Saturday, time unknown. When is pottery?"},
    {"role": "assistant", "content": "Pottery is on Saturday. The record does not give a time."}
  ]
}

Do not add hidden evidence only to the training target. If the correct answer requires information the input does not contain, the example may teach guessing or memorization.

Apply the model's chat template and inspect the resulting tokens. For response-only training, the loss should cover the intended assistant tokens. The exact masking options depend on the trainer and template; verify them on decoded examples instead of trusting a flag name.

Find out which tokens teach the model

Imagine a processed chat containing six user tokens and four assistant tokens. For response-only training, the six user positions supply context but do not contribute target loss; the four intended assistant positions do. Special-token treatment depends on the trainer and template.

Now truncate the sequence before the assistant response. The record may remain a valid list of token IDs. Is it still a useful supervised example for this objective?

For practice, inspect one short example, one long conversation, and one multi-turn record. Write down exactly which responses should be trained. This is more informative than assuming a setting called “assistant only” matches every chat template automatically.

Split by the source of similarity

If conversations from the same underlying document or user appear in both training and evaluation, random row splitting can leak patterns. Group related records before splitting. Keep a final test set outside prompt tuning, training decisions, and checkpoint selection.

Start with a small pilot. Track training loss, validation loss, task outcomes, and examples of changed behavior. A falling loss means the model predicts the training targets more successfully; it does not establish that those targets are useful or that the model generalizes.

Compare before and after on the same cases

# Runnable: Python 3, standard library.
# Fictional paired outcomes for five held-out cases.
baseline = [True, False, True, False, True]
adapted = [True, True, False, True, True]
gains = sum(not a and b for a, b in zip(baseline, adapted))
regressions = sum(a and not b for a, b in zip(baseline, adapted))
assert (gains, regressions) == (2, 1)
print('Gains:', gains, 'Regressions:', regressions)

The adapted model has a net gain, but one previously correct case broke. Inspect it. An average improvement can conceal a failure in an important user journey.

Save the base revision, data version, adapter or full weights, tokenizer, template, training settings, and evaluation results. Test the actual serving format before release; a notebook result does not prove that a merged or quantized deployment behaves identically.

Exercise: validation loss improves while factual accuracy falls. Should you train longer?

Next, learn how a small pair of matrices can adapt a much larger frozen model.

Sources

The current TRL SFT Trainer documentation describes supported dataset formats and loss options. InstructGPT provides a research example of supervised demonstrations within a larger post-training pipeline.

Continue: LoRA (Low-Rank Adaptation).