A model says that 18 people can sit at four tables with four seats each. It also gives a confident paragraph explaining the arrangement. The paragraph sounds reasonable, but the multiplication is wrong for the task. What kind of explanation would actually help you catch that?
Before you begin: Know the difference between a model's answer and evidence supporting that answer.
Expose a checkable calculation
For the seating problem, a useful derivation is short: four tables times four seats gives 16 seats; 18 people need at least two more seats. If only whole four-seat tables are available, five tables provide 20 seats.
Ask for the answer, key assumptions, and a concise calculation you can check. You do not need a transcript of private internal reasoning. The goal is an external justification that connects the inputs to the result.
The distinction matters because an explanation can be generated after or alongside an answer and still contain mistakes. A model's written rationale is not a guaranteed faithful record of every internal computation. Treat it as another artifact to inspect.
Where chain-of-thought prompting came from
The original 2022 chain-of-thought prompting work included demonstrations with intermediate reasoning steps before the final answer. In the models and tasks studied, this improved performance on several reasoning benchmarks. It was an empirical result under particular conditions, not a theorem that adding a phrase improves every model.
Later systems can be trained for reasoning and expose different controls or explanation styles. Some do not expose their internal reasoning. Follow the current model's guidance rather than assuming an older prompt recipe transfers unchanged. A clear task, necessary information, and a verifiable output remain useful across these differences.
Which part of the explanation can you verify without trusting the speaker?
An explanation can be clear and still contain a wrong calculation. Break the workshop total into quantities you can verify: places sold, price per place, refunds and the room fee. This gives you an independent route to the answer. It also reveals whether the question asks for gross sales, net receipts or profit after the room cost.
Ask for the assumptions and a compact calculation that uses those quantities. You do not need an account of every hidden step inside a model. The useful explanation is one you can inspect, reproduce and correct. If a number changes, rerun the arithmetic rather than trusting that the previous conclusion still holds.
Twelve places at 8 GBP produce 96 GBP before refunds.
Refund four places and account for a 15 GBP room fee.
Net receipts are 64 GBP; after the stated room cost, 49 GBP remains.
Decompose only where it helps
For the seating task, separate people, seats per table, total capacity, and rounding. For a policy question, separate the rule, exception, and whether the case meets the exception. For a comparison, decide the criteria before comparing candidates.
This structure helps because each part can be checked. It can also hurt if you prescribe an incorrect decomposition or introduce assumptions the source does not support. “First estimate the missing attendance” is a poor step if the correct behavior is to ask the organizer for that number.
Longer output is not inherently better reasoning. A concise exact answer may be appropriate for a direct lookup. More generated steps create more opportunities for unsupported claims and use additional time and context.
Add an independent check
Use arithmetic or code to verify a numerical result. For a factual answer, inspect the cited source. For a schedule, test the time constraints. Asking the same model “Are you sure?” can trigger a useful revision, but agreement from the same system is weaker than an independent check.
The self-consistency research approach samples multiple reasoning paths and aggregates final answers. That can help on some tasks, but majority agreement is not proof. Several samples can share the same misunderstanding. More attempts also cost more and require a principled way to choose among conflicting results.
Work an example with an exception
An event has 18 attendees. Each table seats four, but one table must hold supplies and cannot seat anyone. How many total tables are needed? Write the capacity calculation before opening the answer.
Check the capacity and the exception
Five seating tables are needed because four provide only 16 seats and five provide 20. Add one separate supplies table, giving six total tables. An answer of five total tables misses the non-seating table. The useful explanation names that assumption explicitly, making the mistake easier to detect.
Now change attendance to 16. The total becomes five tables: four for seating and one for supplies. This nearby case tests whether the method uses the actual constraint rather than repeating the previous answer.
The next lesson combines task patterns into short workflows without turning prompting into a collection of magic phrases.
Practice with feedback
Ask for reasoning you can check independently
Twelve places at 8 GBP, four refunded, plus a 15 GBP room fee. The model answers "Total takings: 96 GBP" with a confident explanation.
Expose a calculation and add a check that does not trust the explanation.
Make the decision before reading the feedback
Check your understanding
Now make something you can check
Write a prompt that separates extracted numbers from the total, so a program can verify the arithmetic.
These notes stay on this page. Download them before leaving.
Check your reasoning against these points
- The output separates each extracted quantity from the final total
- Extracted values are traceable to phrases in the text
- Your check recomputes the total without asking the model
- The failure path is defined rather than left to a human noticing
Compare with a worked answer
Compare the decisions and the evidence. Your wording can be different.
Prompt: "From the text, return only this JSON: {"places_sold": int, "price_each_gbp": number, "refunds": int, "fixed_costs_gbp": number, "total_gbp": number, "working": string} Rules: every number must come from the text. Do not round. 'working' is one line showing the arithmetic."
Model output: {"places_sold": 12, "price_each_gbp": 8, "refunds": 4, "fixed_costs_gbp": 15, "total_gbp": 49, "working": "(12-4)*8 - 15"}
How my code checks the answer: expected = (places_sold - refunds) * price_each_gbp - fixed_costs_gbp assert abs(expected - total_gbp) < 0.005 The model never sees this line, so it cannot talk its way past it.
What I do when the check fails: Discard total_gbp and use the recomputed value; log the case with the raw text. If extraction itself is wrong, the recomputed number is wrong too, so I also flag any case where places_sold < refunds or any value is missing, and send those to a person.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Further reading
Chain-of-Thought Prompting and Self-Consistency describe the historical methods. Language Models Don't Always Say What They Think studies limitations of treating generated explanations as faithful accounts.