Scaling laws: spending a training budget wisely

A fixed training budget buys either more parameters or more training tokens. Which trade should you make? The budget arithmetic can identify possible runs, but only evidence about learning can rank them.

Before you begin: You understand parameters, training tokens, validation loss, and scientific notation.

Scaling laws are empirical relationships between resources and measured outcomes. Researchers fit them to experiments, then use them to estimate promising configurations. They are useful because large training runs are expensive. They are limited because a fitted relationship is not a law of nature.

Keep three quantities separate

Let N be the number of model parameters, D the number of training tokens processed, and C the training computation. For a rough estimate of a conventional dense transformer, people often use C ≈ 6ND floating-point operations.

The factor six is an approximation to forward and backward computation. Attention at long context, embedding details, architecture, recomputation, and other operations can change the estimate. It is not a hardware invoice. Actual time depends on utilization and communication, and actual cost depends on the hardware contract.

# Runnable: Python 3, standard library.
budget = 6e20
for parameters in [1e9, 2e9, 5e9]:
    tokens = budget / (6 * parameters)
    print(f'{parameters / 1e9:.0f}B parameters: '
          f'{tokens / 1e9:.0f}B tokens')

The output is approximately 100, 50, and 20 billion tokens. All three choices use the same simplified budget. Nothing in this calculation tells you their validation losses yet.

Add the missing column

The calculation above produces three feasible configurations. Make a table with parameters, tokens, and held-out loss. Leave the loss column blank until it is measured or estimated by a disclosed fitted model. This empty column is the central uncertainty, not a gap to fill with an attractive number.

Suppose the middle configuration has the lowest measured loss on your pilot runs. Is it enough to declare the same ratio optimal at a hundred times the budget?

Keep training-optimal and serving-optimal decisions in separate columns as well. They can favor different configurations without either calculation being inconsistent.

Fit a relationship, not a slogan

A useful conceptual form is an irreducible loss term plus a model-size term and a data-size term. Each reducible term shrinks with a fitted power of its resource. The coefficients and exponents depend on the experimental setup. Copying constants from a paper into a different tokenizer, dataset, or training recipe can produce precise-looking nonsense.

The 2020 scaling-law study helped establish predictable trends in language-model loss. The 2022 Chinchilla study revisited compute allocation and found that, in its setting, scaling model size and training data together made better use of computation than some earlier large-model recipes.

“About twenty training tokens per parameter” is a historical rule of thumb associated with that work. It is not a required ratio for every model or a stopping rule for production training.

Training cost is only one objective

Suppose a model will serve millions of requests. Training a smaller model on more data can cost extra up front but reduce repeated inference expense. That choice may be sensible even when it is not optimal for minimizing loss at a fixed training compute budget.

The data itself also matters. Repeating the same tokens is not equivalent to collecting equally useful new examples. Leakage into evaluation data can make scaling appear more successful than it is. Lower average next-token loss does not guarantee better factual answers, safer tool actions, or better performance in an underrepresented language.

Make a defensible plan

Run smaller controlled experiments using the intended data mixture and tokenizer. Reserve evaluation data before tuning. Fit the relationship, inspect residual errors, and test a configuration outside the fitting set before relying on a larger extrapolation. Report uncertainty and assumptions alongside the proposed budget.

Exercise: two models have equal validation loss, but one is half the size. Which should you ship?

Next, read the original scaling paper as evidence: what was measured, what was inferred, and what later work changed.

Sources

Compare Scaling Laws for Neural Language Models with Training Compute-Optimal Large Language Models. Their empirical findings inform this lesson; the budget calculation is an illustrative estimate, not a reproduction of either paper.

Continue: Paper: Scaling Laws for Neural Language Models.