DoRA (weight-decomposed low-rank adaptation), explained

DoRA (weight-decomposed low-rank adaptation) is a variant of LoRA that splits each pretrained weight matrix into a magnitude (the length of each column) and a direction (each column normalised to unit length), then fine-tunes the magnitude directly with one trainable scalar per column and the direction with a LoRA update. The fine-tuned weight is $W^{\prime }=m\odot \frac{W_0+BA}{\Vert W_0+BA{\Vert }_c}$. It adds only $k$ parameters per matrix on top of LoRA, merges into a plain weight matrix after training so inference cost is unchanged, and in its paper consistently narrowed the gap between LoRA and full fine-tuning. In the post-training stack it is a drop-in replacement for LoRA under any objective: SFT, DPO, GRPO.

The problem and where DoRA came from#

LoRA usually gets close to full fine-tuning but not all the way, and the reason was not well understood. Liu, Wang, Yin, Molchanov, Wang, Cheng and Chen (2024), at NVIDIA and HKUST, published at ICML 2024, approached the question by measuring how each method changes the weights. Borrowing the reparameterisation from weight normalisation (Salimans and Kingma, 2016), they decomposed every weight matrix into column magnitudes and column directions and tracked, over training checkpoints and layers, the average change in magnitude $\Delta M$ and the average change in direction $\Delta D$ (one minus cosine similarity to the pretrained column).

The pattern was striking. Full fine-tuning showed a negative correlation between the two (-0.62): layers either changed direction a lot with small magnitude changes, or the reverse. LoRA showed a strong positive correlation (0.83): magnitude and direction always moved together, proportionally. The authors' reading is that LoRA's additive low-rank update cannot easily change one without the other, which limits the kinds of adjustment it can make. DoRA reparameterises so that magnitude and direction are separately trainable, and in their measurements it showed a negative correlation (-0.31) like full fine-tuning. On commonsense-reasoning benchmarks DoRA beat LoRA by 3.7 points on LLaMA-7B (78.4 versus 74.7), 1.0 on LLaMA-13B, 2.1 on LLaMA2-7B and 4.4 on LLaMA3-8B (85.2 versus 80.8), and a DoRA variant with half as many trainable parameters as LoRA still beat it, by 2.8 points on LLaMA-7B and 4.2 on LLaMA3-8B.

The math#

Magnitude and direction

Let $W_0\in {\mathbb{R}}^{d\times k}$ be a pretrained weight, with columns $w_{0,j}\in {\mathbb{R}}^d$ for $j=1,\dots ,k$. Define the column-wise norm $\Vert W{\Vert }_c\in {\mathbb{R}}^{1\times k}$ as the vector of column lengths. Every weight matrix can be written as

where $\odot$ scales column $j$ of the normalised matrix by $m_j$. At initialisation $m=\Vert W_0{\Vert }_c$ and $V=W_0$, so $W=W_0$ exactly. DoRA keeps $m$ as a trainable vector and adapts $V$ with LoRA:

Trainable parameters per matrix: $r(d+k)$ for the LoRA factors plus $k$ for the magnitudes. For a $4096\times 4096$ projection at $r=16$ that is $131,072+4,096=135,168$, 3% more than LoRA.

What the gradient does differently

Write $V^{\prime }=W_0+BA$ and let $G={\nabla }_{W^{\prime }}\mathcal{L}$ be the gradient full fine-tuning would see. Per column $j$, the gradients with respect to the direction and magnitude are

The direction gradient is the full gradient for that column, projected onto the plane orthogonal to the current column and scaled by $m_j/\Vert v_j^{\prime }\Vert$. The component of $g_j$ along the column, which would only change its length, is removed from the direction update and routed entirely to the scalar $m_j$. This is the weight-normalisation mechanism: it improves the conditioning of the direction update (the paper argues, following Salimans and Kingma, that scaling and projection bring the gradient's covariance closer to the identity) and decouples "how much" from "which way". In LoRA, both kinds of change have to be expressed through the same $BA$, and the gradients for $B$ and $A$ are just projections of $G$ that mix the two.

A second, simpler way to see DoRA's advantage is expressiveness. Rescaling column $j$ by a factor $s_j$ is the update $\Delta W=W_0(\mathrm{diag}(s)-I)$, which is generally full rank. LoRA at rank $r$ can only approximate it with the top $r$ singular directions of that update. DoRA represents it exactly with $k$ numbers. Fine-tuning updates seem to contain a substantial component of this kind, which is one plausible reason DoRA helps.

Side by side with LoRA and full fine-tuning

The memory trick: treating the norm as a constant

Differentiating through $\Vert W_0+BA{\Vert }_c$ requires storing the full $d\times k$ matrix $V^{\prime }$ for the backward pass in every adapted layer. Liu et al. propose treating the norm as a constant in backpropagation: its value still reflects the current $BA$ in the forward pass, but no gradient flows through it. With that change the direction gradient becomes $\frac{m_j}{\Vert v_j^{\prime }\Vert }{g}_j$, losing the explicit projection, and the authors measured about 24.4% lower training memory on LLaMA-7B with an accuracy difference of only 0.2 points. Common implementations, including the PEFT library's, detach the norm by default.

Worked example: a 2×2 matrix by hand#

Take $W_0=\left(\begin{array}{cc}3& 0\\ 4& 1\end{array}\right)$. Its columns are $(3,4)$ and $(0,1)$, so $m=\Vert W_0{\Vert }_c=(5,1)$ and the directions are $(0.6,0.8)$ and $(0,1)$. Suppose after some training the LoRA product is $BA=\left(\begin{array}{cc}0.5& 0.2\\ 0& 0.2\end{array}\right)$. Then $V^{\prime }=W_0+BA=\left(\begin{array}{cc}3.5& 0.2\\ 4& 1.2\end{array}\right)$ with column norms 5.315 and 1.217.

Under plain LoRA, $W^{\prime }=V^{\prime }$: the first column got 6.3% longer and the second 21.7% longer, as a side effect of the rotation the update was trying to make. Under DoRA with learned magnitudes $m=(5.5,1.0)$, the weight is

with column norms exactly 5.5 and 1.0. The same rotation, but the first column is now 10% longer than the original and the second keeps its original length. Magnitude and direction were chosen independently. (We checked these numbers in PyTorch.)

A toy run: what DoRA can represent that LoRA cannot#

To isolate the expressiveness argument, we trained full fine-tuning, LoRA and DoRA (both at rank 4, with the detached norm for DoRA) to fit a known target weight on a 128×128 linear layer, using mean squared error on 4,096 random inputs, Adam at learning rate 1e-3, batch 256, 2,000 steps, seed 0. We built four targets from the same frozen random $W_0$: an additive rank-4 change $W_0+U{V}^{\top }$; a pure magnitude change (each column rescaled by a factor drawn from $[0.7,1.3]$); a pure direction change (the rank-4 change, with each column renormalised back to its original length); and both together. The reported error is the squared distance to the target, as a fraction of the distance from $W_0$ to the target.

On the additive rank-4 target all three methods reached an error below 0.0001; there DoRA has nothing to add. On the magnitude-only target, LoRA plateaued at 0.839: a column rescaling is full rank, and four directions capture only 16% of it. DoRA fitted it to below 0.0001 using its magnitude vector. On the direction-only target LoRA reached 0.021 (renormalising the columns introduces a small full-rank component) and DoRA below 0.0001. With both changes, LoRA stalled at 0.339 while DoRA and full fine-tuning both converged. In each case LoRA's error was the same at step 500 as at step 2,000: these are representational floors, not slow optimisation.

This toy is built to show the mechanism, and two targets were constructed with DoRA's own parameterisation, so it is not evidence about real fine-tuning. What it does show is the specific class of update DoRA adds for $k$ extra parameters per matrix: independent per-column (per input feature) rescaling. Whether that helps on your task is an empirical question, which the paper's benchmarks answer favourably for commonsense reasoning and visual instruction tuning.

Running the magnitude and direction diagnostic on your own model#

The analysis that motivated DoRA is cheap to reproduce on any fine-tuned checkpoint, and it is a useful diagnostic even if you never switch methods. For a matrix with pretrained columns $w_{0,j}$ and fine-tuned columns $w_j$, compute

for every adapted matrix of one type (say, all query projections) at several checkpoints, and plot $\Delta D$ against $\Delta M$. Liu et al. report that full fine-tuning gives a cloud with a negative slope, LoRA a tight positive line, and DoRA a negative slope closer to full fine-tuning. In our toy run, which ends at the exact targets, LoRA on the combined target finished at $\Delta M=0.027$ and $\Delta D=0.026$, while DoRA and full fine-tuning both reached $\Delta M=0.149$ and $\Delta D=0.028$: LoRA matched the direction change but delivered less than a fifth of the required magnitude change. If the plot for your LoRA run is a tight line and your task metric lags full fine-tuning, that coupling is a candidate explanation and DoRA is a reasonable next experiment. If the LoRA plot already looks like full fine-tuning's, the gap probably lies elsewhere (rank, target modules, learning rate), and DoRA is unlikely to close it.

The same numbers are also a sanity check on any adapter run. Very large $\Delta M$ in a few layers, with tiny $\Delta D$, means those layers are mostly being rescaled; very large $\Delta D$ everywhere means the run has moved far from the base and forgetting is worth measuring.

The algorithm, step by step#

for each target linear layer with frozen W0 (d x k):
    m = column_norms(W0)              # trainable, shape (k,)
    A ~ random (r x k); B = 0 (d x r) # trainable LoRA on the direction
for step in range(total_steps):
    for each adapted layer:
        V = W0 + gamma * B @ A                       # materialised d x k
        norm = column_norms(V).detach()              # constant in backward (memory trick)
        W = V * (m / norm)                           # per-column rescale
        y = x @ W.T
    loss = objective(outputs); loss.backward()       # grads to A, B, m only
    optimizer.step(); optimizer.zero_grad()
after training: W' = m * (W0 + gamma B A) / ||W0 + gamma B A||_c  -> store as an ordinary dense matrix

A note on orientation. The paper writes $W\in {\mathbb{R}}^{d\times k}$ and normalises the $k$ columns; in a PyTorch nn.Linear the weight is stored as (out, in), and implementations normalise over the input dimension so that there is one magnitude per output feature. The two are the same operation under the transpose; just make sure your magnitude vector has as many entries as the dimension you are not summing over.

The data requirements and training loop are otherwise identical to LoRA's. DoRA is used with the same datasets, sizes and objectives. The paper's experiments covered fine-tuning for commonsense reasoning, visual instruction tuning, and image and video-text understanding.

Hyperparameters that matter#

The last row is easy to get wrong. Weight decay on LoRA factors pulls the update toward zero, which means toward the pretrained model. Weight decay on DoRA's magnitudes pulls every column toward zero length, which is a large and unwanted change. If your optimiser applies decay to all trainable parameters, exclude $m$, or decay toward $\Vert W_0{\Vert }_c$ instead. Beyond that, DoRA rarely needs separate tuning: start from a working LoRA configuration, switch it on, and compare at equal steps.

Compute and memory cost#

Parameters and optimiser state grow by $k$ per matrix, which is negligible. The real cost is activations and compute: every adapted layer must materialise the $d\times k$ matrix $W_0+\gamma BA$ and its column norms on every forward pass, instead of computing $W_{0}x+\gamma B(Ax)$ through two thin matmuls. For a $4096\times 14336$ MLP projection that is a 117 MB temporary in bf16, created and discarded per layer per step. With the norm detached, it does not need to be kept for backward, but the extra matmul ($BA$, cost $dkr$) and the normalisation are still there. Expect DoRA steps to be noticeably slower than LoRA steps and peak memory somewhat higher; measure on your configuration. A rule of thumb:

At inference there is no cost at all: $W^{\prime }$ is computed once and stored as an ordinary matrix. If you serve adapters unmerged on a shared base (the multi-adapter pattern that makes LoRA attractive), DoRA is less convenient, because the column-normalised weight cannot be applied as a cheap additive side path; each adapter effectively modifies the full matrix.

Failure modes and how to detect them#

No gain over LoRA

The most common outcome on tasks where LoRA already matches full fine-tuning. If a LoRA run and a full fine-tuning run give the same held-out metric, DoRA cannot do better than both. Run the comparison at equal steps and equal rank, and look at the gap to full fine-tuning rather than the absolute score.

Magnitude drift

Monitor $\Vert m-\Vert W_0{\Vert }_c\Vert /\Vert W_0{\Vert }_c$ per layer. In a healthy run it should stay small and grow slowly. Rapid growth points to a learning rate that is too high for the magnitude parameters, weight decay applied to $m$, or a bug in the norm axis. A model whose column norms have shrunk generates degraded, often repetitive text.

Wrong normalisation axis

Normalising over the wrong dimension still trains, but it is a different method (one magnitude per input instead of per output feature, or vice versa) and the initialisation no longer reproduces $W_0$ exactly if $m$ was computed on the other axis. Check that the model's outputs at step 0 match the base model to numerical precision.

Throughput surprise

A DoRA run that is much slower than its LoRA twin is expected; one that runs out of memory where LoRA did not usually has the norm detaching disabled. Check that setting before reducing batch size.

When to use DoRA, and what to use instead#

Use DoRA when you are committed to a low-rank adapter (for memory, storage or forgetting reasons), there is a measurable gap between your LoRA run and full fine-tuning, and you can afford slower training steps. It is a cheap experiment: one flag in most libraries, the same hyperparameters, and no inference cost after merging. Skip it when LoRA already matches full fine-tuning, when training throughput is the constraint, or when you need to serve many unmerged adapters on one base.

Minimal implementation#

The layer below follows PyTorch's (out, in) weight layout, so the norm is taken over the input dimension and there is one magnitude per output feature. The PEFT library exposes the same method as a LoRA option; in POFE it is a toggle on the LoRA recipe.

import math, torch, torch.nn as nn

class DoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad_(False)
        out_f, in_f = base.weight.shape
        self.A = nn.Parameter(torch.empty(r, in_f)); nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.B = nn.Parameter(torch.zeros(out_f, r))
        self.scale = alpha / r
        with torch.no_grad():
            w = base.weight.float()
            self.m = nn.Parameter(w.norm(dim=1))      # one magnitude per output row (paper's columns of W^T)

    def forward(self, x):
        V = self.base.weight + self.scale * (self.B @ self.A).to(self.base.weight.dtype)
        norm = V.float().norm(dim=1).detach()          # treated as constant in backward
        W = V * (self.m / norm).to(V.dtype).unsqueeze(1)
        y = x @ W.T
        return y + self.base.bias if self.base.bias is not None else y

    @torch.no_grad()
    def merged_weight(self):
        V = self.base.weight.float() + self.scale * self.B @ self.A
        return (V * (self.m / V.norm(dim=1)).unsqueeze(1)).to(self.base.weight.dtype)

# with PEFT:
# from peft import LoraConfig
# cfg = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", use_dora=True)

# exclude magnitudes from weight decay
decay = [p for n, p in model.named_parameters() if p.requires_grad and not n.endswith(".m")]
no_decay = [p for n, p in model.named_parameters() if p.requires_grad and n.endswith(".m")]
opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.01}, {"params": no_decay, "weight_decay": 0.0}], lr=2e-4)

Verify at step 0 that DoRALinear(base)(x) equals base(x) to bf16 precision; that single check catches axis and initialisation bugs.

Further reading#

Liu et al. (2024) is the DoRA paper, with the weight-decomposition analysis, the gradient derivation and the memory ablation. Salimans and Kingma (2016) introduced the magnitude and direction reparameterisation and the conditioning argument behind it. For the adapter DoRA builds on, read LoRA, and for how LoRA's learned solutions differ from full fine-tuning, Shuttleworth et al. (2024) and Biderman et al. (2024).

Frequently asked questions#

References#

1. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arxiv.org/abs/2402.09353
2. Tim Salimans, Diederik P. Kingma (2016). Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. arxiv.org/abs/1602.07868
3. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
4. Reece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha Sharma (2024). LoRA vs Full Fine-tuning: An Illusion of Equivalence. arxiv.org/abs/2410.21228
5. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, et al. (2024). LoRA Learns Less and Forgets Less. arxiv.org/abs/2405.09673

Related reading