LoRA (low-rank adaptation), explained

LoRA (low-rank adaptation) fine-tunes a large model by freezing every pretrained weight matrix and learning, for each chosen matrix, a small additive update constrained to low rank: $W=W_0+\frac{\alpha }{r}BA$, with $B$ and $A$ thin matrices of rank $r$. Only $B$ and $A$ receive gradients and optimiser state, so a 4096×4096 projection that has 16.8 million parameters is adapted with 131 thousand. LoRA is not a training objective. It is a parameterisation that plugs into any objective: SFT, DPO, GRPO, distillation. It sits in the post-training stack as the default way to make those stages fit on modest hardware and to keep many task-specific variants of one base model cheap to store and serve.

The problem and where LoRA came from#

Full fine-tuning updates every parameter. With AdamW in mixed precision that costs about 16 bytes per parameter in weights, gradients and optimiser state, so an 8B model needs around 128 GB before activations, and every fine-tuned variant is a full copy of the model on disk. Before LoRA, parameter-efficient methods either added sequential adapter layers, which add inference latency, or tuned prompt prefixes, which consume context and are hard to optimise.

LoRA was introduced by Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang and Chen (2021). Its motivation came from Aghajanyan, Zettlemoyer and Gupta (2020), who showed that fine-tuning has a low intrinsic dimension: optimising just 200 parameters, randomly projected back into the full space, got RoBERTa to 90% of full fine-tuning performance on MRPC. If the useful update lives in a small subspace, Hu et al. reasoned, one can parameterise the update of each weight matrix directly as low rank. On GPT-3 175B they reported a 10,000-fold reduction in trainable parameters and a threefold reduction in GPU memory relative to Adam fine-tuning, with quality on par or better on their benchmarks, and no added inference latency because the update can be merged into the weights.

The parameterisation, with every symbol defined#

Take a linear layer with frozen pretrained weight $W_0\in {\mathbb{R}}^{d\times k}$ mapping an input $x\in {\mathbb{R}}^k$ to $h\in {\mathbb{R}}^d$. LoRA adds a trainable update of rank at most $r$:

$r$ is the rank, $\alpha$ is a fixed scaling hyperparameter, and $\gamma$ is the scale applied to the update. $A$ is initialised from a small random Gaussian (or Kaiming uniform in common implementations) and $B$ is initialised to zero, so $BA=0$ at step 0 and the adapted model starts exactly at the pretrained model. The number of trainable parameters per matrix is $r(d+k)$ instead of $dk$.

Gradients and why the zero init matters

Let $G=\partial \mathcal{L}/\partial W$ be the gradient full fine-tuning would apply to this matrix (for a batch, $G=\sum \delta x^{\top }$ where $\delta =\partial \mathcal{L}/\partial h$). By the chain rule, LoRA's gradients are projections of $G$:

At initialisation $B=0$, so $A$ gets zero gradient and only $B$ moves, in the direction $G{A}^{\top }$: the full gradient projected onto the random row space of $A$. After the first steps both matrices move and the subspace itself adapts. One first-order step of size $\eta$ on both changes the effective weight by

which shows two things. LoRA's update is the full gradient filtered through the current subspaces $A^{\top }A$ and $B{B}^{\top }$, so it can only follow directions the adapter currently spans. And the effective step scales with ${\gamma }^2$ times the norms of $A$ and $B$, which is why $\alpha$, $r$ and the learning rate are coupled and cannot be tuned independently.

The α/r scale, rsLoRA and LoRA+

Hu et al. divide by $r$ so that, with Adam, changing the rank does not require retuning the learning rate much in their setting. Kalajdzievski (2023) argued that this factor over-shrinks high-rank adapters and that the stable choice as $r$ grows is $\gamma =\alpha /\sqrt{r}$ (rank-stabilised LoRA, rsLoRA), which lets larger ranks actually use their extra capacity. Hayou, Ghosh and Yu (2024) showed with width-scaling arguments that $A$ and $B$ should not share a learning rate; LoRA+ sets ${\eta }_B=\lambda {\eta }_A$ with $\lambda$ around 16, reporting 1 to 2% improvements and up to roughly 2× faster fine-tuning. Both are one-line changes in common libraries.

LoRA next to its siblings

PiSSA (Meng, Wang and Zhang, 2024) keeps LoRA's architecture but initialises $B$ and $A$ from the principal singular components of $W_0$ and freezes the residual, which it reports converges faster; on GSM8K with Mistral-7B it reached 72.86% against LoRA's 67.7%.

Worked example: counting parameters and bytes#

For one $4096\times 4096$ attention projection at $r=16$: $dk=16,777,216$ and $r(d+k)=16\times 8192=131,072$, a ratio of 0.0078. Now a full model. A Llama-style 8B model has 32 layers, hidden size 4096, grouped-query attention with key and value projections of 4096×1024, and an MLP with intermediate size 14,336. Per layer, LoRA on all seven linear maps at $r=16$ adds

Across 32 layers that is 41.9M trainable parameters, about 0.5% of the model. Their AdamW state in fp32 (master weights, two moments) plus bf16 gradients is about $41.9\text{M}\times 14\approx 0.59$ GB. The frozen base in bf16 is about 16 GB. Compare 128 GB for full fine-tuning's weights, gradients and optimiser. The adapter checkpoint in bf16 is about 84 MB, so a hundred task variants take less disk than one extra copy of the base. Activations do not shrink: backpropagation still has to pass gradients through every frozen layer to reach the adapters, so the activation term is the same as for full fine-tuning at the same batch and sequence length.

A toy run: rank sets the floor, scaling sets the speed#

We fitted a known target update in PyTorch to see what rank and scale actually do. The setup is a 128×128 linear layer with a frozen random $W_0$ and a target update $\Delta W^{\star }$ with singular values proportional to $1,\frac{1}{2},\frac{1}{3},\dots$, a heavy-tailed spectrum like the ones measured in fine-tuning updates. The loss is mean squared error on 4,096 random inputs, batch 256, Adam, seed 0, and we report the relative error $\Vert \gamma BA-\Delta W^{\star }{\Vert }_F^2/\Vert \Delta W^{\star }{\Vert }_F^2$. By the Eckart-Young theorem the best any rank-$r$ matrix can do is the energy in the discarded singular values, ${\sum }_{i>r}{\sigma }_i^2/{\sum }_i{\sigma }_i^2$.

At learning rate 2e-3 and $\alpha =16$ with standard $\alpha /r$ scaling, after 1,500 steps LoRA reached 0.3901, 0.1311, 0.0329 and 0.0051 for ranks 1, 4, 16 and 64, against floors of 0.3892, 0.1304, 0.0323 and 0.0047. Gradient descent on $BA$ finds essentially the best low-rank approximation of the update; the remaining error is entirely the rank constraint. That is the real meaning of "LoRA learns less": when the update you need has significant energy outside the top $r$ directions, no amount of training recovers it.

At a lower learning rate (3e-4) the scaling rule starts to matter. With $r=128$ (full rank in this toy), the standard rule gives $\gamma =0.125$ and the error after 500 steps was 0.0337; with rsLoRA's $\gamma =16/\sqrt{128}=1.41$ it was 0.0015, and it closed 90% of the gap in 100 steps rather than 300. At $r=8$ the two rules behaved alike (error 0.1944 and 0.2026 at step 500, both reaching about 0.07 by step 3,000). Small ranks are insensitive to the choice; large ranks under $\alpha /r$ are effectively trained at a much smaller step size, which is the slowdown rsLoRA describes.

Why low rank is usually enough, and when it is not#

The toy makes the trade-off explicit, but it helps to ask what the rank of a useful update actually is in a real model. A fine-tuning run moves weights in response to gradients $G={\sum }_n{\delta }_n{x}_n^{\top }$, sums of outer products of backpropagated errors and inputs. When the training data is narrow (one format, one domain, one style), the errors and inputs that matter span few directions and the accumulated update is dominated by a handful of singular values. This is why Hu et al. found very small ranks (as low as 1 to 8 on $W_q$ and $W_v$) competitive on GPT-3 for tasks like WikiSQL and MultiNLI, and why the intrinsic-dimension results of Aghajanyan et al. hold. The update becomes high rank when the data is broad and the run is long: millions of tokens of code or maths, a new language, or facts the model does not know. Each distinct piece of knowledge needs its own directions, and the spectrum flattens. Biderman et al.'s measurement that full fine-tuning updates have 10 to 100 times higher rank than typical adapters is a measurement of exactly this.

Reinforcement learning sits at the opposite extreme. A policy-gradient step carries, per sampled sequence, a scalar advantage times a log-likelihood gradient, so the information per episode is a few bits. Updates from GRPO or RLVR runs therefore carry little information per step, and LoRA at modest rank is a common choice in open-source RL recipes. Preference optimisation is similar: a DPO run on tens of thousands of pairs rarely needs rank above 32. The practical rule follows: low rank for alignment, style and RL; higher rank or full fine-tuning for knowledge and broad skills.

The second practical consequence is serving. Because the adapted model is $W_0$ plus a small delta, one copy of the base can serve many adapters at once. Open-source inference engines batch requests for different adapters through the same base weights, computing $W_{0}x$ once per token and the per-request $BAx$ with a small gathered matmul. For a product with one model per customer or per task, that is the difference between hosting one model and hosting hundreds.

The algorithm and training loop#

load base model in bf16; freeze all parameters
for each target linear layer (q, k, v, o, gate, up, down):
    add A (r x k) ~ small random, B (d x r) = 0, scale gamma = alpha / r   # or alpha / sqrt(r)
optimizer = AdamW(params = all A and B)               # optional: lr_B = 16 * lr_A (LoRA+)

for step in range(total_steps):
    h = W0 x + gamma * B (A (dropout(x)))             # in every adapted layer
    loss = objective(model outputs)                   # SFT NLL, DPO loss, GRPO surrogate ...
    loss.backward()                                   # grads flow through frozen W0 to reach earlier adapters
    optimizer.step(); optimizer.zero_grad()

after training: W = W0 + gamma * B A   (merge for deployment) or keep the adapter separate

The loop is the loop of whatever objective you use. LoRA changes three things: the parameters handed to the optimiser, the memory profile, and one useful trick for preference and RL training. Because the adapter can be switched off, the frozen base model is the reference policy. DPO and PPO-style methods can compute reference log-probabilities by running the same network with adapters disabled, so no second copy of the model is held in memory.

Data requirements are those of the objective. LoRA does not change how many SFT examples or preference pairs you need. It does change how much the model can absorb: Biderman et al. (2024) found that with roughly 100k instruction pairs or 20B tokens of continued pretraining in code and maths, standard low-rank LoRA substantially underperformed full fine-tuning, and that full fine-tuning's weight perturbations had rank 10 to 100 times higher than typical LoRA configurations. For a few thousand to a few tens of thousands of examples, the gap is usually small.

Hyperparameters that matter#

Tune in this order. First get the learning rate right, since it interacts with $\gamma$ and is the most common cause of a LoRA run that looks like it is doing nothing (too low) or diverges in the first hundred steps (too high). Then rank: double it and see whether the held-out metric moves; if not, you are not rank-limited. Then target modules: excluding the MLP is a common reason LoRA underperforms full fine-tuning on knowledge-heavy tasks.

Compute and memory cost#

Memory, with $P$ base parameters, $P_a$ adapter parameters (usually 0.1 to 2% of $P$) and bf16 storage:

For 8B that is about 17 GB plus activations against 128 GB plus activations. With gradient checkpointing and a moderate sequence length, LoRA SFT of an 8B model fits on one 40 to 80 GB GPU; a 70B model needs about 140 GB of bf16 weights, so two to four 80 GB GPUs, or QLoRA on one. Compute per token is slightly less than full fine-tuning: the forward pass is unchanged (plus a tiny adapter matmul), and the backward pass still computes activation gradients but skips weight gradients for frozen matrices, roughly $4PD$ instead of $6PD$ FLOPs for $D$ tokens. Hu et al. reported about 25% higher training throughput on GPT-3 175B. Do not expect LoRA to be dramatically faster per step; its savings are memory, storage, and the ability to batch many adapters on one serving base.

Failure modes and how to detect them#

Training that barely moves

The loss plateaus near its starting value, adapter weight norms $\Vert B{\Vert }_F$ stay tiny, and outputs look like the base model. Causes: learning rate copied from full fine-tuning (too small by about 10×), a very small $\gamma$ from high rank with $\alpha /r$ scaling, or adapters attached to the wrong module names so nothing is actually trainable. Print the trainable parameter count before every run; it should be what your arithmetic says.

Rank-limited quality

Held-out loss plateaus higher than a full fine-tuning baseline and doubling the rank lowers it. This is the toy run's floor showing up in a real model. Typical on large-scale code or maths, continued pretraining, or teaching facts. Raise the rank, add MLP modules, or switch to full fine-tuning.

Forgetting via intruder dimensions

LoRA forgets less than full fine-tuning on average, but Shuttleworth, Andreas, Torralba and Sharma (2024) found that LoRA-trained matrices acquire new high-ranking singular vectors ("intruder dimensions") that full fine-tuning does not create, that these cause most of LoRA's forgetting, and that they accumulate in continual learning. Watch a general-capability regression suite, especially when stacking several LoRA stages; merging and then training a fresh adapter compounds the effect.

Merge and precision mismatches

The merged model evaluates differently from the unmerged one. This happens when the adapter was trained against a quantised base and merged into a bf16 base (or vice versa), or when merging in low precision rounds away small updates. Evaluate after merging, in the precision you will serve.

When to use LoRA, and what to use instead#

Use LoRA as the default for SFT, DPO and RL on models you cannot comfortably fully fine-tune, when you need many variants of one base, when you want a built-in reference policy, or when forgetting is a bigger risk than under-fitting. Prefer full fine-tuning when you have large datasets that teach new knowledge or skills, when you have the hardware, and when the result will become a new base model for further work.

Minimal implementation#

A LoRA linear layer is a dozen lines in PyTorch. Writing it once makes the shapes, init and scaling concrete; in practice the PEFT library provides the same thing with merging, saving and many adapter variants.

import math, torch, torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.0, rslora=False):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad_(False)
        d, k = base.out_features, base.in_features
        self.A = nn.Parameter(torch.empty(r, k))
        self.B = nn.Parameter(torch.zeros(d, r))          # zero init: model starts at W0
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / math.sqrt(r) if rslora else alpha / r
        self.drop = nn.Dropout(dropout)
        self.enabled = True                               # switch off to get the reference policy

    def forward(self, x):
        y = self.base(x)
        if self.enabled:
            y = y + self.scale * (self.drop(x) @ self.A.T @ self.B.T)
        return y

    @torch.no_grad()
    def merge(self):
        self.base.weight += (self.scale * self.B @ self.A).to(self.base.weight.dtype)
        self.enabled = False

def add_lora(model, targets=("q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"), **kw):
    for name, mod in list(model.named_modules()):
        for child_name, child in list(mod.named_children()):
            if isinstance(child, nn.Linear) and child_name in targets:
                setattr(mod, child_name, LoRALinear(child, **kw))
    return model

# with PEFT:
# from peft import LoraConfig, get_peft_model
# cfg = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", lora_dropout=0.0, use_rslora=False)
# model = get_peft_model(model, cfg); model.print_trainable_parameters()

Pass only parameters with requires_grad to the optimiser. For LoRA+, put the $B$ matrices in a parameter group with 16× the learning rate. In POFE, LoRA is an option on every training recipe rather than a separate method, which reflects what it is: a choice about which parameters move, independent of the loss that moves them.

Further reading#

Hu et al. (2021) is the original paper and still the clearest on the rank-deficiency analysis. Aghajanyan et al. (2020) explains why low-dimensional fine-tuning works at all. Biderman et al. (2024) is the best controlled comparison with full fine-tuning, and Shuttleworth et al. (2024) shows how the learned solutions differ structurally. Kalajdzievski (2023) and Hayou et al. (2024) are the two scaling fixes. Meng et al. (2024) covers PiSSA initialisation, and the QLoRA and DoRA articles cover the two most used extensions.

Frequently asked questions#

References#

1. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
2. Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arxiv.org/abs/2012.13255
3. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, et al. (2024). LoRA Learns Less and Forgets Less. arxiv.org/abs/2405.09673
4. Reece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha Sharma (2024). LoRA vs Full Fine-tuning: An Illusion of Equivalence. arxiv.org/abs/2410.21228
5. Damjan Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arxiv.org/abs/2312.03732
6. Soufiane Hayou, Nikhil Ghosh, Bin Yu (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. arxiv.org/abs/2402.12354
7. Fanxu Meng, Zhaohui Wang, Muhan Zhang (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. arxiv.org/abs/2404.02948
8. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arxiv.org/abs/2305.14314

Related reading