Your model invents a price whenever a record leaves it blank. What did the demonstrations teach it to do with missing evidence? Instruction tuning is as much about designing contrasting cases as collecting fluent target responses.
Before you begin: You understand supervised fine-tuning and chat roles.
Instruction tuning uses demonstrations to shape responses to requests. The lesson in each example is broader than its literal wording. Examples also teach when to answer, what to preserve, how much to explain, and whether uncertainty should be acknowledged.
Write the target before scaling the dataset
Start with a small collection you can inspect thoroughly. For each record, identify the user goal, the information available to the model, and the behavior the response should demonstrate.
For a workshop assistant, include a complete record, a missing time, conflicting versions, an unrelated question, and an unclear request. Keep the difference between the cases visible. If the correct response changes, the input should contain the reason.
An instruction such as “be accurate” is useful only when the demonstrations make accuracy observable. A target that invents a detail contradicts that instruction, however polished the prose is.
Teach distinctions, not a catchphrase
| Input condition | Behavior to demonstrate |
|---|---|
| Evidence answers the question | Give the answer with its support. |
| A required field is absent | Say what is missing. |
| Two sources disagree | Identify the conflict or apply an explicit authority rule. |
| A tool reports failure | Report failure without claiming completion. |
| The request is ambiguous | Ask for the detail needed to proceed. |
Vary wording, length, domain details, and difficulty while preserving the target behavior. Replacing only names in a template creates many rows without much diversity. Include natural user language, not just perfectly formed instructional prompts.
Change the label while preserving the evidence
Prepare two proposed targets for an input that lacks a price. A says “The price is not listed.” B says “The price is probably 20.” Ask a reviewer to identify the exact source supporting each factual claim. B's softer wording does not create evidence for 20.
Now add a real source field price: 20. The target should change because the input changed, not because the reviewer prefers more complete-looking answers.
How would you test the learned distinction?
Hold out new items with both present and absent prices. Check whether the model answers when evidence exists and preserves unknowns when it does not. A model that says “not listed” to everything learned a different shortcut. Evaluate both sides of the decision boundary.
Use this paired design for tool success versus failure, a valid versus expired record, and a clear versus ambiguous request. These contrasts add behavioral variety without padding the dataset with renamed copies of the same easy example.
Inspect what the trainer actually sees
The dataset's visible messages are not the final training sequence. The chat template inserts role markers and special tokens. Tokenization divides the text. Truncation may remove part of the answer. Loss masking decides which tokens are supervised.
Decode several processed examples and inspect the labels. Check that the intended assistant response remains and that at least some response tokens contribute to the loss. In current TRL, assistant-only loss requires a compatible template that marks assistant generation spans. A flag cannot recover those spans from an incompatible template by magic.
Long conversations need special care. If truncation repeatedly drops the final assistant turn, you can waste a run training on the wrong material. If packing combines examples, verify the trainer's attention and boundary behavior rather than assuming all packing methods isolate examples identically.
Review agreement, not just cleanliness
Have reviewers explain disagreements using a rubric. Does a concise answer omit necessary context? Does a detailed answer introduce unsupported facts? Does a refusal block a harmless request? These decisions reveal the behavior being taught.
Keep final evaluation cases separate from demonstration writing and filtering. A dataset assembled to match the test questions may produce a flattering score while generalizing poorly.
Exercise: write two targets for “What time is pottery?” One input says “Saturday, 14:00.” The other says “Saturday, time to be announced.”
Compare your reasoning
The first should state Saturday at 14:00. The second should say the day is known and the time has not been announced. Adding the same cheerful guessed time to both would teach a harmful shortcut. A useful dataset includes both cases and checks that the model distinguishes them.
Next, learn what changes when the supervision is a preference between two responses instead of one demonstration.
Sources
Finetuned Language Models Are Zero-Shot Learners studies instruction tuning across tasks. The current TRL SFT Trainer guide documents assistant-only loss and template requirements.
Continue: RLHF (Reinforcement Learning from Human Feedback).