How much approximation can a model tolerate when its weights use fewer bits? Start with rounding a few numbers, then distinguish the quantization algorithm, numeric format, file container, and hardware implementation.
Before you begin: You understand model weights, numeric precision, and held-out evaluation.
The cost is approximation. Whether that approximation matters depends on the model, tensors, method, hardware, and task. “Four-bit” is a storage description, not a quality score.
Map a range to a smaller set of values
In a simple symmetric integer scheme, choose a scale, divide values by it, round to integers, and multiply by the scale to reconstruct them. Values outside the supported range are clipped.
# Runnable: Python 3, standard library.
weights = [-1.0, -0.2, 0.0, 0.3, 0.9]
scale = max(abs(w) for w in weights) / 127
quantized = [max(-127, min(127, round(w / scale))) for w in weights]
restored = [q * scale for q in quantized]
error = max(abs(a - b) for a, b in zip(weights, restored))
assert error <= scale / 2 + 1e-12
print(quantized, round(error, 6))
The assertion holds here because the chosen range covers the input and rounding is to the nearest level. A production implementation must handle all-zero tensors and its exact rounding convention. Asymmetric schemes can also use a zero point; nonuniform schemes use differently spaced representable values.
Which artifact did you actually evaluate?
A numerical representation change can alter model behavior even when the architecture and nominal parameter count remain the same. Evaluate the quantized artifact and runtime you intend to serve, not only the higher-precision checkpoint from which it was produced.
The uniform scalar experiment shows one small rounding effect with fully visible arithmetic. Real quantization methods use different schemes and can interact with kernels and hardware. The challenge asks you to carry the exact artifact identity into the evaluation so deployment does not silently substitute an untested model representation.
The evaluation uses a higher-precision checkpoint.
Deploy a four-bit artifact to fit the available hardware.
The served representation needs its own quality and performance checks.
Decide what gets quantized
Weight-only quantization reduces stored model weights while other computations may use a higher precision. Other approaches quantize activations, the key/value cache, or several categories. These have different accuracy and hardware implications.
The total memory footprint also includes activations, cache, metadata, buffers, and runtime overhead. A fourfold reduction in raw weight bytes does not guarantee a fourfold reduction in peak request memory.
Per-channel or group-wise scales can represent local ranges more accurately than one scale for a whole tensor, at the cost of more metadata. Outliers can make a global scale waste many levels on rarely used extremes.
Calibration should resemble the workload
Some post-training methods use representative examples to choose or optimize quantization settings. A calibration set containing only short English sentences may not represent long multilingual code tasks. Keep calibration separate from final evaluation.
Quantization-aware training incorporates quantization effects into training. It is different from applying a post-training conversion to an unchanged checkpoint. File containers and runtime formats are also distinct from algorithms: a format name alone does not fully specify how values were quantized.
Decode names that describe different things
GPTQ is a post-training weight-quantization approach that uses calibration activations and an approximation related to second-order error information to reduce reconstruction error. AWQ, activation-aware weight quantization, uses activation information to identify and protect important weight channels through scaling. They are methods, not interchangeable file suffixes.
FP8 denotes eight-bit floating-point formats; exponent and mantissa layouts determine range and precision. It is not the same number system as eight-bit integers with a scale. GGUF is a model file container used in the llama.cpp ecosystem and related tools. A GGUF file can contain different quantization types; the container name alone does not specify a quality or speed level.
Can you compare two files labeled “4-bit” from their sizes alone?
No. Inspect the exact method, group size, metadata, tensors left at higher precision, calibration, and supported runtime. Compare the deployed artifacts on the same task and workload. Similar raw storage can conceal different numerical behavior and kernel performance.
The GPTQ paper and AWQ paper describe their respective methods. The GGUF specification defines the container. These distinctions make the following artifact comparison interpretable.
Compare the artifact you will serve
Measure task quality, structured-output validity, long-context behavior, memory, latency, and throughput. Use the same prompts and generation settings where possible. Check the exact backend: an unsupported low-bit format may require conversions or run more slowly than expected.
Keep the unquantized reference and the conversion configuration. Test edge cases where small numeric differences can change token selection, such as close alternatives or strict tool arguments. A small average numerical error is not a complete behavioral evaluation.
Exercise: a quantized model fits in memory, but long requests still fail. What should you inspect?
Compare your reasoning
Cache and activation memory, batch size, sequence length, and temporary buffers. Weight compression solved one memory category. It did not remove the others.
Practice with feedback
Trade representable levels for rounding error
Place 2^bits equally spaced levels between -1 and 1, including both ends. Round the chosen scalar to the nearest level. The spacing is 2 / (2^bits - 1).
16 levels; absolute error 0.03667
The marked values show the original scalar and its reconstruction. More bits create finer steps. A single small error says little about errors accumulating through a network, so the served artifact still needs evaluation.
What this experiment assumes. Uniform scalar quantization. This is not NF4, a specific inference kernel, or a simulation of whole-model answer quality. Notes and recorded results here last until you leave this page.
Compare the artefact you will actually serve
You evaluate a model at bf16, then deploy a 4-bit build because it fits the GPU.
Map ranges to fewer levels, choose what to quantise, and evaluate the served build.
Check your understanding
Your task
Plan the quantisation comparison on the artefact you will serve.
These notes stay on this page. Download them before leaving.
What to include
- Every row is a build you could actually deploy
- Calibration data is described and justified
- A worst-performing slice is reported, not just an average
- The decision names the tradeoff accepted
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Candidate builds: bf16 (reference), int8 weight-only, 4-bit weight-only with a group size of 128, and 4-bit with a lower group size. Calibration set: 512 real notices sampled across four months and both languages we handle, at production lengths. Not wikitext, which is the default in most examples and looks nothing like our traffic.
| Build | size | p95 latency | all-3-correct | worst slice | |---|---|---|---|---| | bf16 | 14.0 GB | 1.9s | 96% | Welsh notices 88% | | int8 w-only | 7.2 GB | 1.6s | 96% | Welsh 87% | | 4-bit g128 | 4.1 GB | 1.3s | 93% | Welsh 71% | | 4-bit g32 | 4.6 GB | 1.4s | 95% | Welsh 84% |
The slice I check specifically: Welsh notices, 4% of traffic. The average barely moves between bf16 and 4-bit g128 (96 to 93), and that slice drops 17 points. Quantisation damage is rarely uniform, and it concentrates on the rarer patterns, which is exactly where a per-slice number is needed and an average is not.
Decision: 4-bit with group size 32. It fits the card, is faster than bf16, and costs 1 point overall and 4 on the weak slice. I would not take g128 for the 0.5 GB it saves. What would reverse it: if Welsh volume grew, or if the weak-slice figure fell below 80% after a base model upgrade. Both are checked by the same script, which runs on the built artefact, not on the reference weights.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, place the model inside a serving system that controls queues, batching, and request limits.
Sources
The Transformers quantization overview distinguishes supported methods and hardware. QLoRA is one specific use of compact frozen weights during adapter training, not a universal inference recipe.