QLoRA: adapting a model while storing its base weights compactly

LoRA makes the trainable update small, but the frozen base still fills memory. What changes if you store that base in four-bit form? QLoRA combines compact base storage with higher-precision adapter training, keeping storage and computation distinct.

Before you begin: Complete LoRA and understand the difference between a stored weight and a computation using that weight.

The key distinction is between storage precision and computation. Four-bit storage does not mean every multiplication, activation, gradient, and optimizer state uses four bits. Needed base values are reconstructed for computation, while the base remains frozen and gradients update the adapters.

Account for the memory that remains

For seven billion weights, two-byte storage is roughly 14 GB, while raw four-bit storage is roughly 3.5 GB, using decimal units. This is a lower-level weight estimate, not a promise that a complete seven-billion-parameter training run fits in 3.5 GB.

# Runnable: Python 3, standard library.
parameters = 7_000_000_000
for bits in [16, 4]:
    raw_gb = parameters * bits / 8 / 1_000_000_000
    print(bits, 'bits:', raw_gb, 'GB of raw weights')

Quantization needs metadata such as scales. Training also needs adapters, their gradients and optimizer state, activations, and temporary buffers. Long sequences can make activation memory large even when base storage is compact. Measure peak memory on a representative batch before selecting a full run size.

Understand the three named techniques

NF4 is a four-bit representation designed around normally distributed weights. Its representable values are not uniformly spaced. The distributional motivation does not prove that it is best for every tensor or every quantization task.

Double quantization compresses quantization constants as well as the weights they describe. It saves metadata space. It does not mean quantizing every activation twice.

Paged optimizers use memory-management techniques to handle spikes in optimizer memory demand. They can help avoid sudden out-of-memory failures, but transfers and the overall workload still affect performance. They are not a substitute for fitting the run's real working set.

Change one memory contributor at a time

Design a pilot with a fixed base, adapter rank, and microbatch. Measure peak memory at sequence lengths 256, 512, and 1,024, using actual supported configurations. Do not fill the table with estimated benchmark numbers. Then hold sequence length fixed and compare two adapter ranks.

Which pattern would suggest that lowering rank is the wrong first response to an out-of-memory error?

Include a correctness check after each memory-saving change. A run that fits only because truncation removed the required answer is not a successful optimization. The four-bit base estimate is the beginning of memory accounting, not the final training budget.

Read a configuration as a set of hypotheses

The following is an integration configuration, not a complete training script. It requires compatible versions of PyTorch, Transformers, PEFT, and bitsandbytes plus supported hardware. The bfloat16 choice must be supported by that hardware.

# Integration example: requires the packages and hardware named above.
import torch
from transformers import BitsAndBytesConfig
from peft import LoraConfig

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
adapters = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules='all-linear',
    bias='none',
    task_type='CAUSAL_LM',
)

Rank 16 is an experiment, not an optimum. Targeting linear layers broadly follows a common QLoRA-style approach, but the exact modules and exclusions depend on PEFT and the model. Inspect the trainable-parameter list. A narrower adapter may be cheaper; a broader one may capture a useful change that attention-only targets miss.

After loading the quantized base, follow the current PEFT preparation procedure, including prepare_model_for_kbit_training, before attaching trainable adapters. Verify which parameters require gradients. Do not assume an inference-oriented device placement automatically works for distributed training.

Build a run you can diagnose

Start with reviewed examples and a held-out evaluation. Use the deployment chat template and inspect the supervised token mask. Choose sequence lengths from the actual conversation distribution, then test a small batch for memory and numerical stability.

Compare the untouched model, a strong prompt or retrieval baseline, and the adapter. Keep the evaluation inputs and evidence the same. Record task correctness, output validity, useful refusals, general-behavior regressions, peak memory, and serving latency.

Saving an adapter alone is not a complete reproducibility record. Save its base revision, tokenizer, template, quantization and adapter settings, data version, and evaluation protocol. If you merge or requantize for serving, test that exact artifact. The process can introduce additional numerical changes.

Plan an experiment before buying more memory

Exercise: a quantized model loads on your GPU but training fails on long conversations. Should your first move always be to reduce adapter rank?

QLoRA makes useful experiments more accessible. It does not remove the need for good examples or make fine-tuning a dependable database for changing facts.

Sources

QLoRA introduces the storage and training approach. The current PEFT quantization guide documents preparation and adapter configuration. LoRA supplies the underlying low-rank update.

Continue: Paper: LoRA : Low-Rank Adaptation.