LoRA makes the trainable update small, but the frozen base still fills memory. What changes if you store that base in four-bit form? QLoRA combines compact base storage with higher-precision adapter training, keeping storage and computation distinct.
Before you begin: Complete LoRA and understand the difference between a stored weight and a computation using that weight.
The key distinction is between storage precision and computation. Four-bit storage does not mean every multiplication, activation, gradient, and optimizer state uses four bits. Needed base values are reconstructed for computation, while the base remains frozen and gradients update the adapters.
Where training memory goes
For seven billion weights, two-byte storage is roughly 14 GB, while raw four-bit storage is roughly 3.5 GB, using decimal units. This is a lower-level weight estimate, not a promise that a complete seven-billion-parameter training run fits in 3.5 GB.
# Runnable: Python 3, standard library.
parameters = 7_000_000_000
for bits in [16, 4]:
raw_gb = parameters * bits / 8 / 1_000_000_000
print(bits, 'bits:', raw_gb, 'GB of raw weights')
Quantization needs metadata such as scales. Training also needs adapters, their gradients and optimizer state, activations, and temporary buffers. Long sequences can make activation memory large even when base storage is compact. Measure peak memory on a representative batch before selecting a full run size.
What quantizing the base model saves
Quantizing the frozen base reduces one large memory term. Adapter parameters, gradients, optimizer state, activations and temporary work still need memory during training. That is why a weight-file estimate cannot by itself tell you whether a run will fit on a GPU.
The scalar experiment illustrates rounding, not QLoRA’s NF4 representation. Use it to understand why representation changes deserve evaluation, then return to the challenge’s complete memory budget. Separate storage format from the numerical format used for computation, and inspect the actual training configuration rather than assuming every tensor is four-bit.
Seven billion scalar weights at four bits have a 3.5 GB raw payload before overhead.
Start training adapters with long sequences and a nontrivial batch.
Additional state and activations can exhaust a device even when the raw base payload fits.
Storage, computation, and training memory
Begin with the raw payload calculation: seven billion weights times four bits, divided by eight bits per byte, is 3.5 billion bytes. This is decimal GB and excludes quantization metadata and other tensors. It is a useful first term because it is easy to audit. It is not the maximum memory a training run will use.
The stored representation and the arithmetic used to compute with it are different choices. A quantized base can be reconstructed into a suitable compute format for operations while the adapters remain trainable. Saying 'four-bit training' without explaining this can leave the false impression that every activation, gradient and optimizer value uses four bits. Inspect the actual implementation and configuration.
Next vary sequence length while keeping the base artifact fixed. More tokens can increase activation memory even though the weight payload has not changed. Vary the batch separately. If the failure follows these changes, another reduction in adapter rank may have little effect on the dominant term. Measure the categories rather than repeatedly changing the smallest visible setting.
The rounding experiment uses equally spaced scalar levels so the error is easy to see. NF4 uses a different representation, and the experiment must not be read as a reproduction of QLoRA. Its purpose is to make representation error concrete before you evaluate the real quantized base and trained adapter together on held-out task behavior.
NF4, double quantization, and paged optimizers
NF4 is a four-bit representation designed around normally distributed weights. Its representable values are not uniformly spaced. The distributional motivation does not prove that it is best for every tensor or every quantization task.
Double quantization compresses quantization constants as well as the weights they describe. It saves metadata space. It does not mean quantizing every activation twice.
Paged optimizers use memory-management techniques to handle spikes in optimizer memory demand. They can help avoid sudden out-of-memory failures, but transfers and the overall workload still affect performance. They are not a substitute for fitting the run's real working set.
Compare memory costs
Design a pilot with a fixed base, adapter rank, and microbatch. Measure peak memory at sequence lengths 256, 512, and 1,024, using actual supported configurations. Do not fill the table with estimated benchmark numbers. Then hold sequence length fixed and compare two adapter ranks.
Which pattern would suggest that lowering rank is the wrong first response to an out-of-memory error?
Read the shape of the memory measurements
If peak memory changes strongly with sequence length but little with rank, activations or sequence-dependent buffers are plausible contributors. Inspect the profiler and selected attention implementation before concluding the exact cause. The experiment narrows the investigation; it does not separate every allocator and kernel detail by itself.
Include a correctness check after each memory-saving change. A run that fits only because truncation removed the required answer is not a successful optimization. The four-bit base estimate is the beginning of memory accounting, not the final training budget.
Read a QLoRA configuration
The following is an integration configuration, not a complete training script. It requires compatible versions of PyTorch, Transformers, PEFT, and bitsandbytes plus supported hardware. The bfloat16 choice must be supported by that hardware.
# Integration example: requires the packages and hardware named above.
import torch
from transformers import BitsAndBytesConfig
from peft import LoraConfig
quantization = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
adapters = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules='all-linear',
bias='none',
task_type='CAUSAL_LM',
)
Rank 16 is an experiment, not an optimum. Targeting linear layers broadly follows a common QLoRA-style approach, but the exact modules and exclusions depend on PEFT and the model. Inspect the trainable-parameter list. A narrower adapter may be cheaper; a broader one may capture a useful change that attention-only targets miss.
After loading the quantized base, follow the current PEFT preparation procedure, including prepare_model_for_kbit_training, before attaching trainable adapters. Verify which parameters require gradients. Do not assume an inference-oriented device placement automatically works for distributed training.
Record and inspect a training run
Start with reviewed examples and a held-out evaluation. Use the deployment chat template and inspect the supervised token mask. Choose sequence lengths from the actual conversation distribution, then test a small batch for memory and numerical stability.
Compare the untouched model, a strong prompt or retrieval baseline, and the adapter. Keep the evaluation inputs and evidence the same. Record task correctness, output validity, useful refusals, general-behavior regressions, peak memory, and serving latency.
| Symptom | First investigation |
|---|---|
| Loss falls, task quality does not | Inspect targets, evidence, and loss masking. |
| Memory still runs out | Measure activations and sequence length. |
| Training becomes unstable | Inspect dtype support, learning rate, and gradients. |
| Serving differs from training | Compare base revision, template, precision, and merge path. |
Saving an adapter alone is not a complete reproducibility record. Save its base revision, tokenizer, template, quantization and adapter settings, data version, and evaluation protocol. If you merge or requantize for serving, test that exact artifact. The process can introduce additional numerical changes.
Plan a small QLoRA experiment
Exercise: a quantized model loads on your GPU but training fails on long conversations. Should your first move always be to reduce adapter rank?
Compare your reasoning
No. Inspect peak memory, shorten the pilot sequence length or microbatch, and consider supported activation checkpointing. Adapter state may be a small part of the peak. Keep enough realistic long examples in evaluation so a memory workaround does not silently remove an important task.
QLoRA makes useful experiments more accessible. It does not remove the need for good examples or make fine-tuning a dependable database for changing facts.
Practice with feedback
Trade representable levels for rounding error
Place 2^bits equally spaced levels between -1 and 1, including both ends. Round the chosen scalar to the nearest level. The spacing is 2 / (2^bits - 1).
16 levels; absolute error 0.03667
The marked values show the original scalar and its reconstruction. More bits create finer steps. A single small error says little about errors accumulating through a network, so the served artifact still needs evaluation.
What this experiment assumes. Uniform scalar quantization. This is not NF4, a specific inference kernel, or a simulation of whole-model answer quality. Notes and recorded results here last until you leave this page.
Account for the memory that remains
A 7B model in 4-bit uses about 3.5 GB for weights, yet your training run still runs out of memory on a 24 GB card.
Break down training memory and change one contributor at a time.
Check your understanding
Your task
Write the memory budget and the one-variable experiment plan.
These notes stay on this page. Download them before leaving.
What to include
- Each term has a number and a reason
- Activations are identified as the dominant term or shown not to be
- Experiments change exactly one variable each
- Confirmation compares against the previous run, not against an assumption
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Budget for the failing configuration: 7B in 4-bit, LoRA r=64 on all linear layers, batch 4, sequence 2048, no checkpointing, AdamW. 4-bit weights 3.5 adapter params (80M) fp16 0.16 adapter grads fp16 0.16 AdamW state (2 moments, fp32) 0.64 activations, batch 4 x seq 2048 14.5 <- dominant dequant buffers and workspace 1.5 fragmentation allowance 1.5 total 21.9 against 24 GB, and the peak during the backward pass pushes past it. It fits on paper and fails in practice, which is exactly what the fragmentation line is there to remind me of.
One-variable experiments, in order: 1. Enable gradient checkpointing. Predicted: activations fall to roughly 4-5 GB, total near 12 GB, at about 30% slower steps. Nothing else changes. 2. If still tight, halve batch to 2 with accumulation 2. Predicted: activations halve again, effective batch unchanged, step time similar. 3. Only then reduce rank 64 -> 16. Predicted: small memory effect (about 0.7 GB) and a possible quality effect, which is why it is last: it is the only lever here that can change the result, rather than just the cost.
Expected fitting configuration: 4-bit weights, r=64, batch 2, accumulation 2, sequence 2048, checkpointing on, paged AdamW. How I confirm causation: record peak allocated memory and step time for every run, change one flag between runs, and compare against the immediately previous run. If I had turned on checkpointing and dropped rank together and it fit, I would know it fits and nothing else.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Sources
QLoRA introduces the storage and training approach. The current PEFT quantization guide documents preparation and adapter configuration. LoRA supplies the underlying low-rank update.