The assistant uses the wrong format and gives an outdated opening time. Are both defects reasons to fine-tune? They may have different causes, and a small diagnostic experiment can tell you which kind of change to try.
Before you begin: You understand pretraining, prompting, and retrieval-augmented generation.
Transfer learning means reusing knowledge learned in one setting for another. In LLM development, adapting pretrained weights is one form. Prompting and retrieval also reuse a pretrained model, but they do not usually change its weights. Keeping that distinction clear makes experiments easier to interpret.
Diagnose before you train
Take twenty real failures and annotate what prevented a good answer. Was the necessary information absent? Was the instruction unclear? Did the model have evidence but use it incorrectly? Did a tool return an error? Did the output violate a stable format?
Do not assume every error belongs to the model. A broken retrieval filter or truncated prompt can make an excellent model look incapable. Fixing the data path may have more impact than a training run.
| Observed need | A useful first experiment |
|---|---|
| Current, citable facts | Retrieve authoritative evidence. |
| A stable response format | Try clear instructions and structured output. |
| Repeated specialized behavior | Compare supervised adaptation with a strong prompt. |
| Missing domain patterns | Investigate data coverage and appropriate training. |
| Failed external action | Inspect tool arguments, permissions, and results. |
These are starting points, not exclusive categories. A trained model can still need retrieval, and an excellent retrieval system can still need better response behavior.
Supply the missing evidence by hand
Take one failed question and give the baseline model the exact current record in context. Keep the question and response requirements unchanged. If the answer becomes correct, the model has demonstrated at least one instance of the needed behavior when evidence is available.
Now repeat on held-out paraphrases and missing-field cases. What can this experiment support, and what does it leave unresolved?
Interpret the intervention
It supports investigating retrieval or context assembly before weight updates. It does not prove the model will always use evidence correctly, and one successful example is weak evidence of generalization. If the same errors persist with correct context, inspect instructions, output validation, and behavior adaptation as separate next experiments.
Keep an error table with “evidence absent,” “evidence misused,” and “format invalid” as distinct labels. Some cases can have more than one cause. This table makes the adaptation decision a response to observed failures rather than a preference for a particular training method.
Separate weights from context
If you add examples to a prompt, the model conditions on them for that request. If you fine-tune, gradient updates change selected parameters and can affect later requests without those examples being present.
Both can overfit in different ways. A prompt may accidentally work only for familiar examples. A fine-tuned model may memorize training phrases or lose useful general behavior. Evaluate on new inputs that represent the real task, not just shuffled copies of training templates.
Facts can be learned in weights, but weight updates are not a dependable replacement for a current, permission-aware database. You cannot reliably point to the exact training document supporting every generated claim or guarantee that one factual update removes all older associations.
Compare complete systems fairly
Freeze a baseline prompt and evaluation set. Compare the proposed adaptation against that baseline with the same available evidence, tool access, and scoring criteria. Otherwise you may credit training for an improvement caused by a better prompt or more context.
Include the ongoing costs: collecting examples, reviewing labels, training, serving adapters, and monitoring regressions. A small accuracy improvement can be worthwhile for a frequent task, but the decision needs actual measurements.
Design the first experiment
Exercise: support replies are polite but omit a required booking reference. Opening times change weekly. You have 500 reviewed conversations. What would you try first?
Compare your reasoning
Provide current opening times through retrieval or a trusted tool, and enforce the booking-reference requirement through the response schema and validation. Use the conversations to evaluate a prompt baseline. Consider supervised adaptation if the stable behavior still fails often enough to justify the cost. Do not train the weekly timetable into weights as the first solution.
The next lesson turns that decision into a fine-tuning experiment with clear inputs, targets, and stopping evidence.
Sources
LoRA demonstrates parameter-efficient adaptation. Retrieval-Augmented Generation provides a contrasting mechanism for supplying external knowledge. The decision examples here concern an application, not a universal ranking of methods.