LoRA: Fine-Tuning as a Low-Rank Delta

Fine-tuning normally means: take every weight and let gradient descent move it. LoRA changes one thing about that sentence.

frozen:     W                        (d × k, untouched)
trained:    ΔW = (α/r) · B · A       (B: d × r,  A: r × k,  r ≪ d, k)
used:       W + ΔW

The base matrix never moves; a product of two thin matrices is trained beside it and added. Because ΔW is a product through an r-dimensional bottleneck, its rank is at most r — and that is the whole idea, not an implementation detail.

What the rank actually limits. A full fine-tune can move a weight matrix anywhere in its parameter space. LoRA constrains the update to rank at most r, in the modules where adapters were attached — not to a fixed subspace, since A and B both train and the directions are learned, but to no more than r independent ones at a time. That is simultaneously the reason it is cheap (a rank-8 adapter on a 4096×4096 matrix trains 0.4% as many numbers, one in 256) and the reason it is a restriction: if the behaviour you want needs change along more independent directions than r allows, more training time will not get you there. Raising the rank buys capacity and costs parameters; it is not the knob that makes training fast.

Why it fits the memory budget so well. Gradients and optimizer states are needed only for the trained parameters. With Adam that is roughly three extra copies of whatever you train — enormous for a full model, negligible for adapters. The frozen base still has to be stored and run, which is why the next trick exists: QLoRA keeps the base in 4-bit and trains the adapter on top of it, so the memory wall moves from optimizer states to plain weight storage.

Merging, and why people keep adapters unmerged. Since the update is additive, W + ΔW can be folded into the base after training, giving a normal model with no inference overhead at all. Unmerged, the adapter stays a separate file and costs a little latency per forward pass — which people pay for the operationally interesting property: one base model in memory, many task adapters swapped or served side by side, each megabytes rather than gigabytes, depending on rank and how many modules carry one. A full fine-tune gives you another copy of the whole model per task.

Where it tends to fall short. The harder the adaptation leans on change spread across many weights, the more a rank-limited delta on a few modules has to give up: full fine-tuning and continued pretraining keep winning on demanding domain adaptation, and the gap is sensitive to rank, to which modules carry adapters, and to the data and budget — it is a trade-off to measure, not a wall. Narrow behaviour — format, tone, a task's conventions, a schema — is what adapters are reliably good at.

The practical reading of the knobs: rank sets how much change is representable; which modules get adapters sets where change is allowed at all (attention projections only, or the MLP too); and α, applied as α/r, sets how strongly the delta lands — which is why raising r alone changes less than people expect. Sweeping the learning rate while leaving rank at 8 is a common way to conclude that "LoRA cannot learn this" when the real constraint was the rank.

The full landscape — LoRA variants, QLoRA, and when a weight delta is worth it at all — is in Fine-Tuning LLMs: Modern Post-Training Deep Dive and When the Weight Delta Is Worth It.