Back
advanced
Fine-Tuning Techniques

LoRA (Low-Rank Adaptation)

Understand LoRA as parameter-efficient fine-tuning: low-rank adapters, rank, alpha, target modules, and deployment

25 min read· LoRA· PEFT· Fine-Tuning· Low-Rank

LoRA (Low-Rank Adaptation)

LoRA fine-tunes a model by adding small trainable adapter matrices while keeping the base model frozen.

Why LoRA works

Full fine-tuning updates all weights. LoRA updates a small low-rank change.

text
base weights frozen + small adapter update

This makes training cheaper and easier to store.

Key settings

SettingMeaning
rankadapter capacity
alphaadapter scaling
target moduleswhich layers receive adapters
dropoutregularization
mergecombine adapter with base for serving

When to use LoRA

Use it for:

  • style adaptation
  • structured output behavior
  • tool-call patterns
  • domain classification
  • small-team fine-tuning

Do not use it to store changing facts.

Knowledge check

Q1: What stays frozen in LoRA?

The base model weights.

Q2: What does rank control?

Adapter capacity and trainable parameter count.