Back
advanced
Fine-Tuning Techniques

QLoRA: Quantized LoRA

Understand how QLoRA combines low-rank adapters and 4-bit quantization to fine-tune large models with less memory

25 min read· QLoRA· Quantization· LoRA· PEFT

QLoRA: Quantized LoRA

QLoRA makes LoRA cheaper by loading the base model in low precision while training small adapters.

The idea

text
quantized frozen base model + trainable LoRA adapters

The base model uses much less memory. The adapters learn the task.

Why it matters

QLoRA made it practical for more teams to fine-tune larger open models without full enterprise-scale hardware.

Key concepts

ConceptMeaning
4-bit quantizationstore base weights compactly
NF4quantization format used for neural weights
LoRA adaptersmall trainable update
paged optimizerreduces memory spikes

Risks

  • data quality still matters
  • quantization can hurt hard tasks
  • adapter overfitting is possible
  • evals must compare against baseline

Knowledge check

Q1: What is quantized in QLoRA?

The frozen base model weights.

Q2: What is trained?

The LoRA adapter parameters.