Quantization: smaller numbers, measured tradeoffs

How much approximation can a model tolerate when its weights use fewer bits? Start with rounding a few numbers, then distinguish the quantization algorithm, numeric format, file container, and hardware implementation.

Before you begin: You understand model weights, numeric precision, and held-out evaluation.

The cost is approximation. Whether that approximation matters depends on the model, tensors, method, hardware, and task. “Four-bit” is a storage description, not a quality score.

Map a range to a smaller set of values

In a simple symmetric integer scheme, choose a scale, divide values by it, round to integers, and multiply by the scale to reconstruct them. Values outside the supported range are clipped.

# Runnable: Python 3, standard library.
weights = [-1.0, -0.2, 0.0, 0.3, 0.9]
scale = max(abs(w) for w in weights) / 127
quantized = [max(-127, min(127, round(w / scale))) for w in weights]
restored = [q * scale for q in quantized]
error = max(abs(a - b) for a, b in zip(weights, restored))
assert error <= scale / 2 + 1e-12
print(quantized, round(error, 6))

The assertion holds here because the chosen range covers the input and rounding is to the nearest level. A production implementation must handle all-zero tensors and its exact rounding convention. Asymmetric schemes can also use a zero point; nonuniform schemes use differently spaced representable values.

Decide what gets quantized

Weight-only quantization reduces stored model weights while other computations may use a higher precision. Other approaches quantize activations, the key/value cache, or several categories. These have different accuracy and hardware implications.

The total memory footprint also includes activations, cache, metadata, buffers, and runtime overhead. A fourfold reduction in raw weight bytes does not guarantee a fourfold reduction in peak request memory.

Per-channel or group-wise scales can represent local ranges more accurately than one scale for a whole tensor, at the cost of more metadata. Outliers can make a global scale waste many levels on rarely used extremes.

Calibration should resemble the workload

Some post-training methods use representative examples to choose or optimize quantization settings. A calibration set containing only short English sentences may not represent long multilingual code tasks. Keep calibration separate from final evaluation.

Quantization-aware training incorporates quantization effects into training. It is different from applying a post-training conversion to an unchanged checkpoint. File containers and runtime formats are also distinct from algorithms: a format name alone does not fully specify how values were quantized.

Decode names that describe different things

GPTQ is a post-training weight-quantization approach that uses calibration activations and an approximation related to second-order error information to reduce reconstruction error. AWQ, activation-aware weight quantization, uses activation information to identify and protect important weight channels through scaling. They are methods, not interchangeable file suffixes.

FP8 denotes eight-bit floating-point formats; exponent and mantissa layouts determine range and precision. It is not the same number system as eight-bit integers with a scale. GGUF is a model file container used in the llama.cpp ecosystem and related tools. A GGUF file can contain different quantization types; the container name alone does not specify a quality or speed level.

The GPTQ paper and AWQ paper describe their respective methods. The GGUF specification defines the container. These distinctions make the following artifact comparison interpretable.

Compare the artifact you will serve

Measure task quality, structured-output validity, long-context behavior, memory, latency, and throughput. Use the same prompts and generation settings where possible. Check the exact backend: an unsupported low-bit format may require conversions or run more slowly than expected.

Keep the unquantized reference and the conversion configuration. Test edge cases where small numeric differences can change token selection, such as close alternatives or strict tool arguments. A small average numerical error is not a complete behavioral evaluation.

Exercise: a quantized model fits in memory, but long requests still fail. What should you inspect?

Next, place the model inside a serving system that controls queues, batching, and request limits.

Sources

The Transformers quantization overview distinguishes supported methods and hardware. QLoRA is one specific use of compact frozen weights during adapter training, not a universal inference recipe.

Continue: Model Serving: vLLM, TGI, SGLang.