How fine-tuning works

A clear prompt still fails on a repeated, well-defined task. Can reviewed demonstrations teach the missing behavior? Fine-tuning is an experiment about that behavior, so the first artifact should be a target you can inspect.

Before you begin: Complete transfer learning and understand training, validation, and test splits.

Fine-tuning continues training from an existing model. Full fine-tuning updates all or most model parameters; adapter methods update a smaller set. In both cases, the objective and examples determine what the model is encouraged to do.

Define the behavior to improve

Avoid “make the model smarter.” A useful goal might be: given a request and a verified workshop record, produce the requested fields, preserve the record's values, and mark missing information as unknown.

Create a baseline evaluation before collecting more training examples. Include ordinary requests, paraphrases, missing fields, contradictory evidence, and out-of-scope questions. Decide how to measure correctness separately from formatting.

Count independent training examples

Write a complete training record

A chat example should include the instructions and context available at deployment. Here is a fictional record, shown as data rather than an API call.

{
  "messages": [
    {"role": "system", "content": "Use the supplied record. Do not invent a time."},
    {"role": "user", "content": "Record: pottery, Saturday, time unknown. When is pottery?"},
    {"role": "assistant", "content": "Pottery is on Saturday. The record does not give a time."}
  ]
}

Do not add hidden evidence only to the training target. If the correct answer requires information the input does not contain, the example may teach guessing or memorization.

Apply the model's chat template and inspect the resulting tokens. For response-only training, the loss should cover the intended assistant tokens. The exact masking options depend on the trainer and template; verify them on decoded examples instead of trusting a flag name.

Which tokens contribute to the loss?

Imagine a processed chat containing six user tokens and four assistant tokens. For response-only training, the six user positions supply context but do not contribute target loss; the four intended assistant positions do. Special-token treatment depends on the trainer and template.

Now truncate the sequence before the assistant response. The record may remain a valid list of token IDs. Is it still a useful supervised example for this objective?

For practice, inspect one short example, one long conversation, and one multi-turn record. Write down exactly which responses should be trained. This is more informative than assuming a setting called “assistant only” matches every chat template automatically.

Keep related examples in the same split

If conversations from the same underlying document or user appear in both training and evaluation, random row splitting can leak patterns. Group related records before splitting. Keep a final test set outside prompt tuning, training decisions, and checkpoint selection.

Start with a small pilot. Track training loss, validation loss, task outcomes, and examples of changed behavior. A falling loss means the model predicts the training targets more successfully; it does not establish that those targets are useful or that the model generalizes.

Compare the original and fine-tuned models

# Runnable: Python 3, standard library.
# Fictional paired outcomes for five held-out cases.
baseline = [True, False, True, False, True]
adapted = [True, True, False, True, True]
gains = sum(not a and b for a, b in zip(baseline, adapted))
regressions = sum(a and not b for a, b in zip(baseline, adapted))
assert (gains, regressions) == (2, 1)
print('Gains:', gains, 'Regressions:', regressions)

The adapted model has a net gain, but one previously correct case broke. Inspect it. An average improvement can conceal a failure in an important user journey.

Save the base revision, data version, adapter or full weights, tokenizer, template, training settings, and evaluation results. Test the actual serving format before release; a notebook result does not prove that a merged or quantized deployment behaves identically.

Exercise: validation loss improves while factual accuracy falls. Should you train longer?

Practice with feedback

import random
from collections import defaultdict

def split_by_origin(examples, test_frac=0.2, seed=0):
    groups = defaultdict(list)
    for e in examples:
        groups[e['origin_id']].append(e)
    ids = sorted(groups)
    random.Random(seed).shuffle(ids)
    cut = int(len(ids) * (1 - test_frac))
    train_ids, test_ids = set(ids[:cut]), set(ids[cut:])
    assert not (train_ids & test_ids)
    train = [e for i in train_ids for e in groups[i]]
    test = [e for i in test_ids for e in groups[i]]
    assert not ({e['origin_id'] for e in train} & {e['origin_id'] for e in test})
    return train, test


def build_labels(tok, prompt, response):
    p = tok(prompt, add_special_tokens=False)['input_ids']
    r = tok(response, add_special_tokens=False)['input_ids'] + [tok.eos_token_id]
    input_ids = p + r
    labels = [-100] * len(p) + r          # only the response is supervised
    return input_ids, labels

ids, labels = build_labels(tok, PROMPT, RESPONSE)
supervised = [i for i, l in zip(ids, labels) if l != -100]
print(repr(tok.decode(supervised)))
# Printing this once caught two bugs in my file: the chat template was adding
# a leading newline that was being supervised, and the EOS token was missing,
# so the model never learned to stop and rambled into a second answer.
#
# On the split: random row splitting gave 0.91 on test. Splitting by origin_id
# gave 0.68. The second number is the real one; the first was measuring
# whether the model could recognise a paraphrase of something it had seen.

Next, learn how a small pair of matrices can adapt a much larger frozen model.

Sources

The current TRL SFT Trainer documentation describes supported dataset formats and loss options. InstructGPT provides a research example of supervised demonstrations within a larger post-training pipeline.