What are model scaling laws?

A fixed training budget buys either more parameters or more training tokens. Which trade should you make? The budget arithmetic can identify possible runs, but only evidence about learning can rank them.

Before you begin: You understand parameters, training tokens, validation loss, and scientific notation.

Scaling laws are empirical relationships between resources and measured outcomes. Researchers fit them to experiments, then use them to estimate promising configurations. They are useful because large training runs are expensive. They are limited because a fitted relationship is not a law of nature.

Model size, training tokens, and compute

Let N be the number of model parameters, D the number of training tokens processed, and C the training computation. For a rough estimate of a conventional dense transformer, people often use C ≈ 6ND floating-point operations.

The factor six is an approximation to forward and backward computation. Attention at long context, embedding details, architecture, recomputation, and other operations can change the estimate. It is not a hardware invoice. Actual time depends on utilization and communication, and actual cost depends on the hardware contract.

# Runnable: Python 3, standard library.
budget = 6e20
for parameters in [1e9, 2e9, 5e9]:
    tokens = budget / (6 * parameters)
    print(f'{parameters / 1e9:.0f}B parameters: '
          f'{tokens / 1e9:.0f}B tokens')

The output is approximately 100, 50, and 20 billion tokens. All three choices use the same simplified budget. Nothing in this calculation tells you their validation losses yet.

The tradeoff between model size and data

Include the training token count

The calculation above produces three feasible configurations. Make a table with parameters, tokens, and held-out loss. Leave the loss column blank until it is measured or estimated by a disclosed fitted model. This empty column is the central uncertainty, not a gap to fill with an attractive number.

Suppose the middle configuration has the lowest measured loss on your pilot runs. Is it enough to declare the same ratio optimal at a hundred times the budget?

Keep training-optimal and serving-optimal decisions in separate columns as well. They can favor different configurations without either calculation being inconsistent.

How to read a scaling relationship

A useful conceptual form is an irreducible loss term plus a model-size term and a data-size term. Each reducible term shrinks with a fitted power of its resource. The coefficients and exponents depend on the experimental setup. Copying constants from a paper into a different tokenizer, dataset, or training recipe can produce precise-looking nonsense.

The 2020 scaling-law study helped establish predictable trends in language-model loss. The 2022 Chinchilla study revisited compute allocation and found that, in its setting, scaling model size and training data together made better use of computation than some earlier large-model recipes.

“About twenty training tokens per parameter” is a historical rule of thumb associated with that work. It is not a required ratio for every model or a stopping rule for production training.

Training cost and inference cost

Suppose a model will serve millions of requests. Training a smaller model on more data can cost extra up front but reduce repeated inference expense. That choice may be sensible even when it is not optimal for minimizing loss at a fixed training compute budget.

The data itself also matters. Repeating the same tokens is not equivalent to collecting equally useful new examples. Leakage into evaluation data can make scaling appear more successful than it is. Lower average next-token loss does not guarantee better factual answers, safer tool actions, or better performance in an underrepresented language.

Plan a training budget

Run smaller controlled experiments using the intended data mixture and tokenizer. Reserve evaluation data before tuning. Fit the relationship, inspect residual errors, and test a configuration outside the fitting set before relying on a larger extrapolation. Report uncertainty and assumptions alongside the proposed budget.

Exercise: two models have equal validation loss, but one is half the size. Which should you ship?

Practice with feedback

C = 1e21          # training FLOPs
SERVE_TOKENS = 5e12   # expected tokens served over the model's life

print(f"{'N':>10} {'D tokens':>12} {'D/N':>7} {'serve FLOPs':>12} {'total':>12}")
for N in (3e8, 1e9, 3e9, 7e9, 2e10):
    D = C / (6 * N)
    serve = 2 * N * SERVE_TOKENS       # ~2 FLOPs per parameter per token
    print(f'{N:10.0e} {D:12.2e} {D/N:7.0f} {serve:12.2e} {C+serve:12.2e}')

#          N     D tokens     D/N  serve FLOPs        total
#      3e+08     5.56e+11    1852     3.00e+21     4.00e+21
#      1e+09     1.67e+11     167     1.00e+22     1.10e+22
#      3e+09     5.56e+10      19     3.00e+22     3.10e+22
#      7e+09     2.38e+10       3     7.00e+22     7.10e+22
#      2e+10     8.33e+09       0     2.00e+23     2.00e+23
#
# Two readings. On training loss alone, compute-optimal for this budget sits
# near D/N around 20, so about 3e9 parameters on 5.6e10 tokens. But at 5e12
# served tokens, inference dominates training by an order of magnitude, and
# the 3e8 model's total is eight times cheaper than the 3e9 model's.
#
# Recommendation: train around 1e9 parameters and spend the remaining budget
# on more tokens rather than more parameters, accepting slightly higher
# training loss for much lower lifetime cost.
# What would change it: if served volume were 5e10 tokens rather than 5e12,
# inference stops dominating and the compute-optimal 3e9 model wins. The
# serving estimate is the assumption doing the real work here, so it is the
# number I would verify first.

Next, read the original scaling paper as evidence: what was measured, what was inferred, and what later work changed.

Sources

Compare Scaling Laws for Neural Language Models with Training Compute-Optimal Large Language Models. Their empirical findings inform this lesson; the budget calculation is an illustrative estimate, not a reproduction of either paper.