Quantization (GPTQ, AWQ, GGUF, FP8)
Quantization stores or computes model values with fewer bits.
Instead of full precision everywhere, you use formats like INT8, INT4, FP8, or GGUF variants to reduce memory and improve speed.
Why it matters
| Benefit | What it unlocks |
|---|---|
| lower memory | fit larger models on available hardware |
| lower bandwidth | faster inference in some settings |
| lower cost | serve more requests per machine |
| local deployment | run smaller models on laptops or edge devices |
Tradeoff
Quantization is compression. Compression can lose information.
Watch for:
- worse reasoning
- worse math/code
- format mistakes
- longer-tail quality drops
- degraded multilingual behavior
Common formats
| Method | Typical use |
|---|---|
| INT8 | safer compression with modest savings |
| INT4 | bigger savings, more quality risk |
| GPTQ | post-training quantization for transformers |
| AWQ | activation-aware quantization |
| GGUF | local inference ecosystem format |
| FP8 | modern accelerator-friendly low precision |
Evaluation rule
Do not judge quantization by one prompt. Run task evals before and after quantization.
Measure:
- quality
- latency
- memory
- throughput
- cost
- failure cases
Knowledge check
Q1: What is quantization trying to reduce?
Memory, bandwidth, and serving cost.
Q2: Why can quantized models fail subtly?
They may pass simple prompts but lose quality on hard or rare cases.