PiSSA (principal singular values and vectors adaptation), explained

PiSSA (Principal Singular values and Singular vectors Adaptation) is LoRA with a different starting point. LoRA adds a zero-initialised low-rank update to a frozen weight; PiSSA instead takes the singular value decomposition of each pretrained weight, puts its top-$r$ singular components into the trainable low-rank factors, and freezes the remainder as a residual matrix. The model computes exactly the same function at step 0, the architecture and inference path are identical to LoRA, and the adapters merge into the weights the same way. What changes is which part of the weight moves: PiSSA fine-tunes the most important directions of each matrix directly, while LoRA starts from nothing and has to discover its directions. The paper reports faster convergence and better results than LoRA across a dozen models, and a quantised variant, QPiSSA, that loses less to 4-bit quantisation than QLoRA. PiSSA is a parameterisation, not an objective; it is used under SFT, DPO or RL losses exactly where LoRA would be.

The problem and where it came from#

LoRA's initialisation sets $B=0$ and $A$ to small random noise so that training starts at the pretrained model. A consequence, visible in the LoRA gradients, is that $A$ receives no gradient on the first step and $B$ receives the full gradient projected onto a random subspace. The first stretch of training is spent rotating that random subspace toward useful directions, with small updates. Meng, Wang and Zhang (2024, Peking University) asked whether a better starting subspace was already available, and argued that the principal singular subspace of the pretrained weight is the natural choice: those directions carry most of the matrix's energy, and fine-tuning them resembles fine-tuning the whole matrix.

They compared PiSSA and LoRA on 12 models from 184M to 70B parameters across five generation and eight understanding tasks and found PiSSA better in their setups. On GSM8K, Mistral-7B fine-tuned with PiSSA reached 72.86% against LoRA's 67.7%; with 4-bit bases, LLaMA-3-70B reached 86.05% with QPiSSA against 81.73% with QLoRA. On GLUE with DeBERTa-v3-base PiSSA won 7 of 8 tasks. Initialisation uses a randomised fast SVD (Halko, Martinsson and Tropp, 2009) and takes seconds per model.

The parameterisation, with every symbol defined#

Use the LoRA article's convention: a frozen weight $W_0\in {\mathbb{R}}^{d\times k}$ maps $x\in {\mathbb{R}}^k$ to $h\in {\mathbb{R}}^d$, and the adapter is $BA$ with $B\in {\mathbb{R}}^{d\times r}$, $A\in {\mathbb{R}}^{r\times k}$. (The PiSSA paper names the factors the other way round, $A\in {\mathbb{R}}^{m\times r}$ and $B\in {\mathbb{R}}^{r\times n}$.) Write the SVD $W_0=U\Sigma V^{\top }$ with singular values ${\sigma }_1\ge {\sigma }_2\ge \dots$. PiSSA splits it at rank $r$:

so that $W_0=W^{\text{res}}+BA$ exactly. During training $W^{\text{res}}$ is frozen and only $B$ and $A$ receive gradients:

The square-root split gives $B$ and $A$ equal norms, $\Vert B{\Vert }_F^2=\Vert A{\Vert }_F^2={\sum }_{i\le r}{\sigma }_i$, which balances their gradients. There is no $\alpha /r$ factor in the paper's setup (it sets lora_alpha equal to the rank, so the scale is 1).

Gradients: why PiSSA moves faster from step 0

With $G=\partial \mathcal{L}/\partial W$ the full-matrix gradient, the adapter gradients are the same expressions as in LoRA:

and one first-order step of size $\eta$ on both factors changes the effective weight by

Compare LoRA at initialisation, where $B=0$ leaves only $-\eta G{A}^{\top }A$ with a random $A$ of small norm. PiSSA's first step is the gradient filtered through the principal row and column subspaces of $W_0$ and weighted by the singular values, so it is both larger and aimed at the directions that matter most for the layer's output. The flip side is that the effective step size is now proportional to ${\sigma }_i$, which is why PiSSA wants a much smaller learning rate than LoRA (the paper used 2e-5).

QPiSSA: quantise the residual, not the whole matrix

QLoRA quantises $W_0$ to NF4 and trains a zero-initialised adapter, so its initial error is all of the quantisation error. QPiSSA quantises only the residual:

where $\Vert \cdot {\Vert }_{\ast }$ is the nuclear norm, the sum of singular values. Removing the top components narrows the residual's value distribution and removes many large entries, so 4-bit blockwise quantisation has smaller steps to make. The paper reports about a 20% reduction across LLaMA layers and up to 49% for key projections of LLaMA-3-70B.

Next to its siblings

MiLoRA (Wang et al., 2024) is the deliberate opposite: it adapts the minor singular components and freezes the principal ones, arguing this preserves pretrained knowledge better. That both directions have been reported to beat LoRA is a hint that initialisation effects depend heavily on the learning rate and task, a point the toy run below makes concrete.

Worked example: the first step on a 2 × 2 matrix#

Let $W_0=\mathrm{diag}(3,1)$, so $U=V=I$, $\sigma =(3,1)$, and take $r=1$. PiSSA sets $B=(\sqrt{3},0{)}^{\top }$, $A=(\sqrt{3},0)$, so $BA=\mathrm{diag}(3,0)$ and $W^{\text{res}}=\mathrm{diag}(0,1)$. Suppose the full gradient at step 0 is $G=\left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right)$. Then $G{A}^{\top }A=\left(\begin{array}{cc}3& 0\\ 3& 0\end{array}\right)$ and $B{B}^{\top }G=\left(\begin{array}{cc}3& 3\\ 0& 0\end{array}\right)$, and one SGD step of size $\eta$ on both factors changes the weight by approximately

LoRA with $B=0$ and, say, $A=(0.6,0.2)$ gets $\partial \mathcal{L}/\partial A=0$ and $\partial \mathcal{L}/\partial B=G{A}^{\top }=(0.8,0.8{)}^{\top }$, so

Full fine-tuning would move by $-\eta G$, norm $2\eta$. At the same learning rate PiSSA's first step is about ten times LoRA's and almost four times full fine-tuning's, and it is concentrated on the entry tied to the large singular value. That is the "faster convergence" mechanism, and it is also a warning: copy a LoRA learning rate into a PiSSA run and the first steps can be far too large.

Toy runs on a real weight matrix#

We took layer 12 of Qwen2.5-0.5B-Instruct and ran three experiments in PyTorch (seeded). First, the spectrum: the $896\times 896$ query projection has top singular values 2.39, 2.35, 2.18, 2.14 and a median of 0.32, and its top 16 components hold 17.9% of the squared Frobenius norm (46.6% for the top 64, 66.7% for the top 128). The MLP matrices are flatter: the $4864\times 896$ gate projection's top 16 hold 8.8%, its top 128 hold 37.5%. Real LLM weights are not low rank, so $W^{\text{res}}$ keeps most of each matrix.

Second, quantisation error. We implemented blockwise NF4 with 64-value blocks (the QLoRA codebook) and measured the nuclear-norm error of quantising $W_0$ against quantising $W^{\text{res}}$.

The reduction grew with the rank: for the query projection 8.8% at $r=16$, 25.8% at $r=64$ and 41.0% at $r=128$; for the gate and down projections 4.5% to 21.1% and 3.9% to 19.3%. Removing just 16 components cut the down projection's largest absolute entry from 0.440 to 0.249, which is why blockwise absmax quantisation benefits even when little energy is removed. These magnitudes are in line with the paper's "about 20%" at the ranks it reports.

Third, convergence. We asked a rank-16 adapter on that query projection to learn a target change $\Delta W^{\star }$ by regression (4,096 Gaussian inputs, batch 256, Adam, 600 steps, best of four learning rates per method), in two versions with equal norm: a change inside the top-16 singular subspace (a random rotation and rescaling of the top components) and a random rank-16 change in arbitrary directions. We report $\Vert \Delta W_{\text{learned}}-\Delta W^{\star }{\Vert }_F^2/\Vert \Delta W^{\star }{\Vert }_F^2$.

When the target lived in the principal subspace, PiSSA got there faster: 0.0024 after 100 steps against LoRA's 0.0189, with both below $10^{-3}$ by step 200. When the target was in random directions, PiSSA could not represent it at all and stalled at 0.914, while LoRA reached 0.0022 by step 200 and essentially zero by 600. The reason is structural. LoRA's reachable set is $W_0+\{\text{rank}\le r\}$; PiSSA's is $W^{\text{res}}+\{\text{rank}\le r\}$. To add an arbitrary rank-16 change, PiSSA's adapter would have to hold $BA+\Delta W^{\star }$, which has rank 32; the best rank-16 matrix leaves 0.901 of the target unexplained, and the run converged to within 0.013 of that floor. At step 0 the gradient norms also differed as the worked example predicts: $\Vert \partial \mathcal{L}/\partial B\Vert$ was 0.0234 for PiSSA and 0.0043 for LoRA, and $\partial \mathcal{L}/\partial A$ was 0.048 for PiSSA and exactly zero for LoRA.

Both cases are caricatures, but they bracket reality. If the fine-tuning update concentrates in the principal subspace, PiSSA has a head start. If it does not, PiSSA must spend capacity overwriting the principal components it was handed. Which regime a real task falls into is an empirical question, which is the honest reading of PiSSA and MiLoRA both reporting wins.

Finally, initialisation time. A full SVD of these matrices took 0.45 to 1.6 seconds on a loaded CPU. Randomised SVD with 4 subspace iterations took 0.01 to 0.04 seconds and captured 17.72% of the query projection's energy at rank 16 against 17.93% for the exact SVD (17.01% with 2 iterations, 17.93% with 16). Because the top singular values are close together, the fast and exact rank-16 subspaces differ noticeably in direction even when they capture almost the same energy, so if you need reproducible adapters across machines, use more iterations or the exact SVD.

The algorithm, step by step#

for each target linear layer W0 (d x k):
    U, S, Vh = svd(W0)                       # or randomized SVD, niter = 4 to 16
    B = U[:, :r] * sqrt(S[:r])               # d x r
    A = sqrt(S[:r])[:, None] * Vh[:r]        # r x k
    W_res = W0 - B @ A                       # frozen; optionally quantise to NF4 (QPiSSA)
    store W_res in place of W0; register B, A as trainable

train exactly as LoRA (same objective, same loop), with a smaller learning rate

after training, either:
    merge:   W = W_res + B' A'               # identical serving to a fully fine-tuned model
    or share as a LoRA on the ORIGINAL W0:   dW = B' A' - B A = [B', B] @ [A'; -A]   # rank 2r adapter

The last line matters operationally. A trained PiSSA adapter only makes sense on top of $W^{\text{res}}$, not on top of $W_0$. To share it as a plug-in adapter for the unmodified base model, convert it to $\Delta W=B^{\prime }{A}^{\prime }-BA$, which the paper writes as a rank-$2r$ LoRA with factors $[B^{\prime }B]$ and $[A^{\prime };-A]$. PEFT supports this conversion at save time. The doubled rank doubles adapter storage, a small cost.

Data needs are those of the objective. The paper's main experiments used 100K-example subsets of MetaMathQA, CodeFeedback and WizardLM-Evol-Instruct for one epoch, and full MetaMathQA-395K for three epochs as a robustness check.

Hyperparameters that matter#

One fairness note when reading comparisons: the paper ran LoRA and PiSSA at the same learning rate of 2e-5 with $\alpha =r$, which is a low rate for LoRA. Since PiSSA's steps are effectively larger at a given rate (see the worked example), some of a same-learning-rate gap can be a step-size effect. Tune the learning rate separately for each method before deciding.

Compute and memory cost#

Training cost is identical to LoRA once initialised: the same $r(d+k)$ trainable parameters per matrix, the same optimiser state (about 16 bytes per trainable parameter in mixed precision), the same activations, and the same backward pass through frozen weights. The frozen part is $W^{\text{res}}$ in place of $W_0$, same size. Memory therefore follows the LoRA rule $M\approx 2P+16P_a+M_{\text{act}}$, or about $0.5P+16P_a+M_{\text{act}}$ for QPiSSA with NF4. Initialisation adds one SVD per adapted matrix: seconds with randomised SVD on a GPU for a 7B model. A full SVD of every matrix of a 70B model is more expensive, and the residual must be written out or recomputed each time the model is loaded, unless you cache the PiSSA-initialised base. Serving cost is zero after merging; as an unmerged LoRA on the original base, the converted adapter has rank $2r$.

Failure modes and how to detect them#

Learning rate too high

Loss spikes in the first tens of steps, or the model's general capability (a held-out perplexity on generic text) degrades sharply early. PiSSA updates the directions that carry most of each layer's output, so overshooting damages the model faster than with LoRA. Lower the learning rate by 3 to 10 times relative to your LoRA setting and use warmup.

Forgetting through the principal directions

Because the trainable part is the model's most important subspace, large updates can erode general capabilities more than LoRA's do. Run a general regression suite alongside the task metric. MiLoRA's motivation is exactly this risk.

Adapter applied to the wrong base

Loading a raw PiSSA adapter ($B^{\prime },A^{\prime }$) on top of an unmodified $W_0$ double-counts the principal components, and outputs degrade badly. Either ship the residual model, or convert to the rank-$2r$ $\Delta W$ form before sharing. Test the shared artefact on a clean base model before releasing it.

Non-reproducible initialisation

Randomised SVD gives slightly different residuals on different machines or seeds; an adapter trained on one residual is subtly wrong on another. Fix the seed and iteration count, or save the residual weights.

When to use PiSSA, and what to use instead#

Minimal implementation#

A PiSSA linear layer in PyTorch, written to make the split explicit. PEFT implements it as a LoRA initialisation option.

import torch, torch.nn as nn

class PiSSALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, niter=None):
        super().__init__()
        W = base.weight.data.float()                          # d x k
        if niter is None:
            U, S, Vh = torch.linalg.svd(W, full_matrices=False)
        else:                                                 # randomized SVD (Halko et al.)
            U, S, V = torch.svd_lowrank(W, q=r + 10, niter=niter); Vh = V.T
        U, S, Vh = U[:, :r], S[:r], Vh[:r]
        self.B = nn.Parameter(U * S.sqrt())                   # d x r
        self.A = nn.Parameter(S.sqrt()[:, None] * Vh)         # r x k
        res = W - self.B.data @ self.A.data                   # frozen residual
        self.register_buffer("W_res", res.to(base.weight.dtype))
        self.bias = base.bias
        self.register_buffer("B0", self.B.data.clone()); self.register_buffer("A0", self.A.data.clone())

    def forward(self, x):
        y = x @ self.W_res.T + (x @ self.A.T.to(x.dtype)) @ self.B.T.to(x.dtype)
        return y if self.bias is None else y + self.bias

    @torch.no_grad()
    def as_lora_delta(self):                                  # share on the ORIGINAL base: rank 2r
        B2 = torch.cat([self.B, self.B0], dim=1)              # d x 2r
        A2 = torch.cat([self.A, -self.A0], dim=0)             # 2r x k
        return B2, A2                                         # W0 + B2 @ A2 == W_res + B @ A

# with PEFT:
# from peft import LoraConfig, get_peft_model
# cfg = LoraConfig(r=16, lora_alpha=16, target_modules="all-linear", lora_dropout=0.0,
#                  init_lora_weights="pissa_niter_16")
# model = get_peft_model(model, cfg)          # lr around 2e-5 to start, then sweep

In POFE, PiSSA is an initialisation switch on LoRA recipes rather than a separate method, and the recipe stores the residual base alongside the adapter so the two cannot be mismatched.

Further reading#

Meng, Wang and Zhang (2024) is the paper; the appendices cover fast SVD, the rank-$2r$ conversion and per-layer quantisation errors. Hu et al. (2021) is the LoRA baseline, and Dettmers et al. (2023) the QLoRA setup that QPiSSA modifies. Li et al. (2023, LoftQ) is the other main approach to quantisation-aware LoRA initialisation. Wang et al. (2024, MiLoRA) argues for the opposite end of the spectrum. Halko, Martinsson and Tropp (2009) is the randomised SVD behind fast initialisation.

Frequently asked questions#

References#

1. Fanxu Meng, Zhaohui Wang, Muhan Zhang (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. arxiv.org/abs/2404.02948
2. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
3. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arxiv.org/abs/2305.14314
4. Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, Tuo Zhao (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models. arxiv.org/abs/2310.08659
5. Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, Yun Chen (2024). MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. arxiv.org/abs/2406.09044
6. Nathan Halko, Per-Gunnar Martinsson, Joel A. Tropp (2009). Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. arxiv.org/abs/0909.4061

Related reading