LoRA: learning a small update to a large model

Could a useful model update depend on just a few combinations of input features? LoRA restricts the learned correction to a small intermediate space while preserving the much larger frozen weight matrix.

Before you begin: You understand fine-tuning and matrix multiplication. Rank is introduced below.

Think of the base model as the starting solution. The adapter learns a correction to selected linear operations. It does not attach a searchable notebook of facts, and a small adapter is not automatically sufficient for every task.

Follow the dimensions

Let a frozen matrix W map input_width features to output_width features. LoRA represents an update as B @ A, where A has shape (rank, input_width) and B has shape (output_width, rank).

y = W @ x + scale * B @ A @ x
standard LoRA scale = alpha / rank

The update has rank at most the chosen rank. The original matrix can still be full rank. LoRA constrains the update, not the entire model's representation.

For a 4,096 by 4,096 matrix and rank eight, full training would update 16,777,216 weights in that matrix. The two adapter matrices contain 8 * (4096 + 4096) = 65,536 parameters, ignoring optional biases. That is a parameter-count calculation, not the same ratio for total training memory.

What does rank mean here?

Rank counts the number of independent directions a linear map can produce. In a rank-one update, every output correction is a multiple of one direction. In the example below, that direction is [0.5, 0.25]; the scalar multiplier is the input difference measured by A.

Predict the correction for x = [2, 2], then for x = [4, 2]. The first has difference zero and therefore no correction. The second has difference two, so its correction is [1, 0.5], just like the original [3, 1] input.

Run those two inputs through the program. The concrete limitation explains both the parameter saving and the capacity tradeoff behind the matrix shapes.

Work through one correction

# Runnable: Python 3, standard library.
def matvec(matrix, vector):
    return [sum(a * b for a, b in zip(row, vector))
            for row in matrix]

W = [[1.0, 0.0], [0.0, 1.0]]
A = [[1.0, -1.0]]  # Rank one: compress two features to one.
B = [[0.5], [0.25]]  # Expand the correction back to two.
x = [3.0, 1.0]
base = matvec(W, x)
update = matvec(B, matvec(A, x))
output = [a + b for a, b in zip(base, update)]
assert output == [4.0, 1.5]
print(output)

The adapter first measures the difference between the two input features, then distributes a scaled correction to both outputs. The values are hand-chosen to expose the operation. A real training run learns them from gradients.

Initialization and scaling matter

A common initialization sets one adapter matrix randomly and the other to zero, so the initial update is zero. Setting both to zero would block useful initial gradients through their product. Different implementations can use other initialization schemes, so inspect the configuration.

Increasing rank adds update capacity and trainable parameters. It may help a difficult adaptation, but it may also overfit or do little when labels are poor. Alpha, rank, initialization, and learning rate interact. Some variants use a different scaling rule, so record the actual method rather than assuming all adapters use alpha / rank.

Target modules decide where the update is allowed. Attention projections are common targets; feed-forward projections can matter too. Module names differ across architectures, including fused projections. Verify the trainable-parameter list before launching a run.

Deploy the adapter deliberately

An adapter checkpoint needs the compatible base model, tokenizer, template, and configuration. Some deployments keep adapters separate; others merge the update into base weights. Merging can remove separate adapter computation, but quantization and dtype conversions may change results and need evaluation.

Exercise: training memory remains high after switching to LoRA. What might still dominate?

Next, combine adapters with compact storage of the frozen base weights.

Sources

LoRA: Low-Rank Adaptation of Large Language Models defines the method. The PEFT LoRA reference documents implementation options and variants; use the reference matching your installed version.

Continue: QLoRA: Quantized LoRA.