A smooth line extends beyond the largest experiment in a paper. Which part did the authors measure, and which part did they predict? Reading a scaling-law result begins at that boundary.
Before you begin: Complete the scaling-laws lesson and distinguish a fitted curve from a measured data point.
The 2020 paper by Kaplan and colleagues asks how language-model performance changes with model size, data, and compute. Read it as an experimental study of a particular family of models and training conditions. Its influence is substantial; its conclusions still have a scope.
Start with what the vertical axis means
The central outcome is language-model loss on held-out text. Loss measures how much probability the model assigns to the observed next tokens. It is not directly a percentage of questions answered correctly, a measure of honesty, or a score for an agent completing tasks.
Even loss comparisons require care. Different tokenizers divide the same text differently, so per-token loss values are not automatically comparable across tokenizations. Different evaluation distributions can change the result too.
Before interpreting a figure, write down its axes, which variables change, which are held fixed, and whether every point comes from a completed training run. That one-minute habit prevents many mistaken summaries.
What does the line in the figure actually support?
A nearly straight line on log-log axes suggests a power-law relationship over the plotted range. Read the axes, fitted quantity and experimental conditions before interpreting the slope. Loss, compute and downstream accuracy are different quantities, and one cannot be silently substituted for another.
Use the budget experiment to separate a model of resource use from a fitted model of performance. Then inspect where the paper has measurements and where a claim becomes extrapolation. The challenge asks you to reconstruct the supported claim in words that preserve those limits.
Measured training runs lie near a fitted line within a particular compute range.
Extend that line far beyond the largest measured run.
The extension is a prediction whose assumptions need testing, not another observation.
Understand a power law in plain terms
If a reducible component of loss follows A * N ** (-alpha), multiplying model size by a fixed factor multiplies that component by another fixed factor. It describes diminishing returns, not a sudden point where improvement must stop.
Here is a made-up exponent to demonstrate the arithmetic. These numbers are not fitted coefficients from the paper.
# Runnable: Python 3, standard library.
alpha = 0.1 # Illustrative, not a reported estimate.
baseline = 1.0
after_doubling = baseline * 2 ** (-alpha)
after_tenfold = baseline * 10 ** (-alpha)
assert 0 < after_tenfold < after_doubling < baseline
print(round(after_doubling, 3), round(after_tenfold, 3))
The small exponent means a large resource increase may bring a modest reduction in that component. If an irreducible term exists, the total loss shrinks by a smaller fraction still. Do not apply the same percentage directly to task accuracy.
Reconstruct a claim without inventing a result
Choose one figure and label every visible quantity as a measurement, a fitted value, or an extrapolation. If the caption does not tell you how a point was produced, consult the methods section before assigning a category.
Then write two statements: “The reported runs show…” and “The fitted relationship suggests…”. Keep the second conditional. For example, a trend in held-out loss may suggest a promising larger run; it does not establish a future model's factual accuracy or deployment cost.
What if the line fits the training points perfectly?
That alone does not validate extrapolation. A flexible model can fit observed points while predicting poorly outside their range. Look for held-out configurations, sensitivity to the fitting procedure, and uncertainty. Also check whether several points share the same underlying training run rather than representing independent experiments.
Use this annotation beside the later comparison with Chinchilla. It helps you explain a changed compute-allocation conclusion through experimental choices instead of describing research as a contest of slogans.
Read the allocation claim with its assumptions
The study found regular patterns and proposed compute-efficient allocation strategies under its setup. Later, Hoffmann and colleagues revisited the balance between model size and training tokens with a different experimental approach. That is a useful scientific correction, not a reason to discard every earlier observation.
Compare the training horizons, learning-rate schedules, model sizes, data, and fitting methods. A model trained with a schedule intended for a much longer run may look worse when stopped early. Experimental design influences the estimated optimum.
Do a paper-reading exercise
Choose one figure from the original paper. Create a note with four parts: the exact claim you believe the figure supports; the conditions under which it was measured; one plausible alternative explanation; and one experiment that could distinguish the explanations.
For example, an observed trend over model sizes might motivate a larger run. A useful follow-up holds data quality and evaluation constant while testing a size outside the fit. An unhelpful follow-up changes the tokenizer, dataset, and architecture simultaneously and then attributes every difference to size.
Knowledge check: can you reproduce the paper by drawing a power-law curve from approximate constants found in a blog?
Compare your reasoning
No. That illustrates a mathematical shape. Reproduction needs the stated experimental setup or the original measurements and fitting procedure, with deviations disclosed. Keep illustrative plots clearly labeled and never invent benchmark tables to make a lesson look complete.
The next architecture lessons apply this reading discipline to model families whose names cover several different designs.
Practice with feedback
Spend a fixed compute budget twice
Use the rough dense-training relation C = 6ND in consistent relative units. N is model size; D is training-token budget. Here D = C / (6N).
Compute stays 300; model size changes from 5 to 10.
At fixed C, doubling N halves D. The figure exposes a budget constraint, not an optimal allocation. A fitted loss model, data quality and operational goals would be needed to compare the resulting trained models.
What this experiment assumes. A teaching approximation for dense-model training. It excludes architecture-specific work and does not predict downstream accuracy or recommend a universal token-to-parameter ratio. Notes and recorded results here last until you leave this page.
Reconstruct a scaling claim without overclaiming
A figure shows loss against compute on log-log axes, with points falling near a straight line.
Read log-log axes, state a power law plainly, and keep conclusions inside the evidence.
Make the decision before reading the feedback
Check your understanding
Now make something you can check
Write the claim, its conditions, and the extrapolation you refuse to make.
These notes stay on this page. Download them before leaving.
Check your reasoning against these points
- The plain sentence expresses a multiplicative relationship
- The prediction is bounded by the studied range and includes numbers
- At least three conditions are named
- The refused extrapolation is specific
Compare with a worked answer
Compare the decisions and the evidence. Your wording can be different.
Axis meanings: horizontal is training compute in FLOPs on a log scale; vertical is test cross-entropy loss in nats per token, also log scale. Both are per-token averages on a held-out split of the same distribution used for training. The relationship in one plain sentence: each time compute is multiplied by ten, the reducible part of test loss falls by a roughly constant factor, until it approaches an irreducible floor. A prediction inside the range: at 1e20 FLOPs the fitted curve gives loss about 2.4; at 1e21 it gives about 2.2. So a 10x budget increase should buy roughly 0.2 nats on this data distribution.
Conditions attached: - Same data distribution and tokenizer as the study. - Same architecture family, with the hyperparameter tuning the authors applied at each scale. - Compute measured the same way, and inside the studied range.
An extrapolation I decline: predicting the loss at 1e26 FLOPs, four orders of magnitude beyond the largest fitted point. The fit is an empirical description over a range, not a law of nature, and the flattening at the right edge is exactly where a departure would first appear. I also decline to translate 0.2 nats into 'better at answering member questions'. Loss on a general distribution is not our task metric, and the mapping between them is not something this figure measures.
How I would test it cheaply on our setup: train three small models at 1e17, 1e18 and 1e19 FLOPs on our own data with identical recipes, plot our own three points, and see whether the slope resembles the published one. Three points is enough to tell 'roughly the same behaviour' from 'our data behaves differently', which is the only question I actually need answered.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Sources
Read Kaplan et al., Scaling Laws for Neural Language Models alongside Hoffmann et al., Training Compute-Optimal Large Language Models. Compare their evidence directly before adopting either as a planning rule.