GaLore (gradient low-rank projection), explained

GaLore (Gradient Low-Rank Projection) is a memory-saving way to train every weight of a model. Instead of constraining the weights to a low-rank update, as LoRA does, it constrains the optimiser state: each weight matrix's gradient is projected onto a small subspace found by an SVD of a recent gradient, Adam's moment estimates live in that subspace, and the resulting update is projected back to full size and applied to the full weight. Every few hundred steps the subspace is recomputed, so over a run the weights can move in any direction. The weights themselves are ordinary dense matrices with no adapter to merge. In the post-training stack GaLore is an optimiser choice for full-parameter SFT or continued pretraining on hardware too small for AdamW's state, and it competes with LoRA, QLoRA and 8-bit optimisers rather than with any training objective.

The problem and where it came from#

Mixed-precision AdamW keeps, per parameter, a weight, a gradient and two fp32 moments, plus usually an fp32 master copy. The two moments alone are 8 bytes per parameter, so for an 8B model they cost 64 GB before anything else. LoRA removes that cost by training only small adapters, but it changes the problem: the update is confined to a rank-$r$ subspace chosen at initialisation, which hurts in pretraining and in fine-tuning runs that need to move many directions (see the LoRA article's discussion of Biderman et al. 2024). ReLoRA periodically merges and restarts adapters to escape this, at the cost of warm starts and resets.

Zhao, Zhang, Chen, Wang, Anandkumar and Tian (2024, ICML) observed that the object worth compressing is the gradient, not the weight. For a linear layer the gradient is a sum of outer products of backpropagated errors and inputs, and they showed that for reversible networks it becomes low rank during training. GaLore keeps the weights full rank and runs the optimiser on a projected gradient. They reported up to 65.5% lower optimiser-state memory with pretraining quality close to full-rank Adam on LLaMA models from 60M to 7B parameters on C4, and, with 8-bit GaLore and per-layer updates, the first pretraining of a 7B model on a single 24 GB consumer GPU without model parallelism, checkpointing or offloading (22.0 GB at a small per-GPU token batch).

The update rule, with every symbol defined#

Take one weight matrix $W_t\in {\mathbb{R}}^{m\times n}$ with $m\le n$ (transpose the roles otherwise) and its stochastic gradient $G_t={\nabla }_W{\phi }_t(W_t)$. GaLore keeps a projector $P_t\in {\mathbb{R}}^{m\times r}$ with orthonormal columns, the top-$r$ left singular vectors of the gradient at the most recent refresh:

The gradient is compressed to $R_t=P_t^{\top }{G}_t\in {\mathbb{R}}^{r\times n}$, and Adam runs entirely in that space:

The normalised low-rank step is lifted back and applied to the full weight with learning rate $\eta$ and a scale $\alpha$:

$T$ is the subspace change frequency, $r$ the rank, $\alpha$ a scale factor that, unlike LoRA's $\alpha /r$, does not depend on $r$. Between refreshes each step moves $W$ only within the column space of $P_t$, a rank-$r$ change. Across refreshes the changes add up, $W_T=W_0+\Delta W_{T_1}+\Delta W_{T_2}+\dots$, and the sum can have any rank. That is the whole trick: low-rank state, full-rank trajectory. When $r=\min (m,n)$ and $P$ is square orthonormal, $P{P}^{\top }=I$ and the algorithm is exactly Adam up to a rotation of the moment coordinates.

Why the gradient is low rank

For a linear layer $h=Wx$, a batch of $B$ token positions gives $G={\sum }_{i=1}^B{\delta }_i{x}_i^{\top }$, where ${\delta }_i$ is the backpropagated error at position $i$. So $\mathrm{rank}G\le B$ trivially, but the useful claim is stronger: the errors and inputs that dominate the sum concentrate in a few directions, and those directions persist across steps. The paper proves (Lemma 3.3) that for a family of reversible networks trained with SGD the gradient's stable rank decreases over training, and gives a convergence result for GaLore with a fixed projector (Theorem 3.8). The convergence result covers progress inside the subspace; it does not promise that a fixed subspace is good enough overall, which is why $P$ is refreshed.

Next to LoRA and full Adam

The table is in numbers of values, following the paper's Table 1. GaLore also needs a full-size gradient for the matrix at the moment it is projected, which the per-layer update trick below keeps from accumulating across layers.

Worked example: one projected step by hand#

Take a $2\times 3$ weight with gradient

so the left singular vectors are $e_1$ and $e_2$ with singular values $\sqrt{10}=3.162$ and $1$. With $r=1$, $P=(1,0{)}^{\top }$ and $R=P^{\top }G=(3,0,1)$, which keeps $10/11=90.9\%$ of the gradient's squared norm. On the first Adam step with bias correction, $\hat{M}=R$ and $\hat{V}=R^{\odot 2}$, so $N=R/(|R|+ϵ)\approx (1,0,1)$: Adam's sign-like first step. The full-size update is $PN=\left(\begin{array}{ccc}1& 0& 1\\ 0& 0& 0\end{array}\right)$, scaled by $\eta \alpha$. Row 2 does not move this step. If later gradients are dominated by row 2, the next SVD refresh puts $e_2$ into $P$ and row 2 starts to train. Full Adam would have stored 12 moment values here; GaLore stores 6 (two $1\times 3$ moments) plus the 2-value projector. The proportion becomes dramatic at scale: for a $4096\times 14336$ MLP matrix at $r=512$, Adam's moments are 117.4M values and GaLore's state is 16.8M.

For a whole Llama-style 8B model (32 layers, width 4096, key-value width 1024, MLP width 14,336), the seven linear maps per layer hold 6.98B parameters. Adam's moments for them are 13.96B values; GaLore's state is 0.58B at $r=128$, 2.32B at $r=512$ and 4.63B at $r=1024$, or 4.1%, 16.6% and 33.2% of Adam's. The embeddings and output head (1.05B parameters for a 128K vocabulary) are usually left on ordinary Adam, as in the paper, and so are norms and biases.

A toy run: subspace switching is what makes it full rank#

We fitted a full-rank target with a single $256\times 256$ linear layer in PyTorch (seed 0): target singular values ${\sigma }_i\propto i^{-1/2}$, random orthogonal singular vectors, 8,192 Gaussian inputs, batch 256, mean-squared error, weights starting at zero. We report the relative error $\Vert W-W^{\star }{\Vert }_F^2/\Vert W^{\star }{\Vert }_F^2$ and, for each method, the best of three learning rates (1e-3, 3e-3, 1e-2). The best any rank-16 matrix can do on this target is 0.448 (Eckart-Young).

Full-rank Adam reached $5\times 10^{-5}$ within 100 steps. A trained low-rank factorisation $BA$ with $r=16$, which is what LoRA on a zero base amounts to, converged to 0.450 and stayed there, against the 0.448 floor. GaLore with the same rank-16 state behaved completely differently once the projector was allowed to move: with $T=200$ it was at 0.355 after 500 steps, 0.251 after 1,000 and 0.013 after 3,000; with $T=50$ it was at 0.129, 0.0011 and below $10^{-4}$. With the projector computed once and never refreshed it stalled at 0.527, worse than the rank-16 optimum, because the first gradient's top subspace is not the best subspace for the final answer. Optimiser state for this layer was 131,072 values for Adam, 12,288 for GaLore and 24,576 (adapters plus moments) for the low-rank factorisation. Two lessons transfer. The refresh is not a detail, it is the mechanism. And on a single well-conditioned problem a shorter $T$ is better; the paper's choice of 200 (with 50 to 1,000 making "no much difference" in their pretraining runs) reflects the noise and cost of SVDs on real gradients.

A probe on real gradients

We also took one backward pass of Qwen2.5-0.5B-Instruct on four short documents (109 tokens) and decomposed the gradients of four matrices. The top 8 singular directions held 88% of the gradient energy in layer 12's $896\times 896$ query projection, 76% in its $4864\times 896$ gate projection and 66% in its down projection; the top 32 held 94% to 99%. With only 109 tokens the rank is at most 109, so those numbers say more about concentration than about rank. The more informative test is transfer: the top-8 subspace from this batch captured 54% of the energy of a second, unrelated batch's gradient for the query projection (a random 8-dimensional subspace captures 0.9%), but only 31% for the gate projection, 5% for the down projection and 4% for layer 2's output projection. At $r=128$ those rose to 86%, 49%, 24% and 19%. Some layers have stable gradient subspaces and some do not, at least at tiny batch sizes. That is consistent with GaLore working well at ranks of hundreds rather than tens, and with the subspace needing periodic refreshes.

The algorithm and training loop#

for each 2-D weight W in attention and MLP layers:       # embeddings, norms, biases: plain AdamW
    state: P (m x r) or Q (n x r), M, V (r x n or m x r), step t
for step in range(total_steps):
    loss = objective(model(batch)); loss.backward()
    for W in galore_params:                                 # or inside a backward hook, layer by layer
        G = W.grad
        if t % T == 0: P = top_r_left_singular_vectors(G)   # SVD of an m x n matrix every T steps
        R = P.T @ G
        M = b1*M + (1-b1)*R;  V = b2*V + (1-b2)*R*R
        N = bias_corrected(M) / (sqrt(bias_corrected(V)) + eps)
        W -= lr * (alpha * (P @ N) + weight_decay * W)
        W.grad = None                                       # per-layer update frees the gradient immediately
    other_params.step()

Two engineering pieces matter. With per-layer weight updates the optimiser step for each matrix runs as soon as its gradient is ready during backpropagation, so the full model's gradients never exist at once; this is where much of the headline memory saving for 7B on 24 GB comes from, together with 8-bit moments from Dettmers et al.'s 8-bit optimisers. And the SVD cost is amortised: one decomposition per matrix every $T$ steps, which the paper put at under 10% overhead. Data requirements are whatever the objective needs. GaLore was designed for and validated mainly on pretraining-scale token counts; for fine-tuning the paper reports GLUE with RoBERTa-base (average 85.89 against LoRA's 85.61 at rank 4) and instruction data with Gemma-2B, Phi-2 and LLaMA-7B at rank 128, where test perplexities were within a few hundredths of full fine-tuning and LoRA (for LLaMA-7B on OpenAssistant: 2.95 GaLore, 2.94 LoRA, 2.98 full).

Hyperparameters that matter#

Compute and memory cost#

Per parameter in mixed precision, full AdamW costs about 2 (bf16 weight) + 2 (bf16 gradient) + 4 (fp32 master) + 8 (moments) = 16 bytes. GaLore removes most of the 8 bytes of moments for projected matrices, replacing them with $4(mr+2nr)/(mn)$ bytes per parameter, and the per-layer update removes most of the gradient term. A rough rule for a model with $P$ parameters, almost all in projected matrices:

Activation memory is unchanged from full fine-tuning, because every weight is trained and backpropagation is ordinary. Compute is full fine-tuning compute (about $6PD$ FLOPs for $D$ tokens) plus the projections, which are $O(mnr)$ per matrix per step, plus an SVD every $T$ steps. In practice GaLore steps are somewhat slower than AdamW steps, and much slower than LoRA when LoRA's frozen base lets it skip weight gradients. GaLore's advantage is never speed; it is reaching full-parameter quality in a memory budget where AdamW does not fit. Compared with LoRA on the same GPU, GaLore uses more memory (dense weights must be stored in trainable precision and the gradient of each matrix must be materialised) and gives a full-rank model as output.

Three practical consequences follow from that output being a dense model. Checkpoints are full model copies, so a hundred task variants cost a hundred times the base on disk, where LoRA would cost a hundred small adapters. Preference stages lose LoRA's free reference model: DPO or PPO-style training after a GaLore SFT needs a separate frozen copy for reference log-probabilities, 2 bytes per parameter more. And distributed training shards differently: optimiser state is already small, so ZeRO-style partitioning of state buys less, while the dense gradient is the term worth sharding. None of these is a reason to avoid GaLore, but they decide whether it or a high-rank LoRA is the cheaper way to get the same quality.

Failure modes and how to detect them#

Stale subspace

Loss improves in bursts right after each refresh and flattens in between, a sawtooth with period $T$. The subspace is going stale faster than it is refreshed. Lower $T$ or raise $r$. Logging the fraction of gradient norm captured, $\Vert P^{\top }G{\Vert }_F/\Vert G{\Vert }_F$, per matrix makes this visible directly.

Moment mismatch after a refresh

The moments $M,V$ are expressed in the old basis; after $P$ changes they are applied to new directions. Small loss spikes right after refreshes are the symptom. The original method accepts this mismatch. If the spikes are large, lower the learning rate or lengthen $T$ so refreshes are rarer, and check that the spikes do not grow over the run.

Rank too low for the task

A gap to full AdamW that does not shrink with more steps, most visible in pretraining-scale runs. The paper's table shows the gap closing as $r/d_{\text{model}}$ rises (on 1B, perplexity 15.64 at $r=512$ against 15.56 for full rank). Double $r$ and compare.

Tooling incompatibilities

Per-layer updates conflict with gradient accumulation and with standard data-parallel gradient all-reduce: the Transformers implementation of the layer-wise variant raises errors for both. Gradient clipping by global norm also needs the full gradient. Decide early whether you need layer-wise updates; without them, GaLore still saves the moments but not the gradient memory.

When to use GaLore, and what to use instead#

Minimal implementation#

A self-contained GaLore-AdamW for 2-D parameters, close to the reference algorithm. The authors' galore-torch package provides GaLoreAdamW, GaLoreAdamW8bit and GaLoreAdafactor with the same parameter-group keys, and Hugging Face Transformers exposes them as optim="galore_adamw" with optim_target_modules.

import torch

class GaLoreAdamW(torch.optim.Optimizer):
    def __init__(self, params, lr=1e-2, rank=128, update_proj_gap=200, scale=0.25,
                 betas=(0.9, 0.999), eps=1e-8, weight_decay=0.0):
        super().__init__(params, dict(lr=lr, rank=rank, gap=update_proj_gap, scale=scale,
                                      betas=betas, eps=eps, wd=weight_decay))

    @torch.no_grad()
    def step(self):
        for g in self.param_groups:
            b1, b2 = g["betas"]
            for W in g["params"]:
                if W.grad is None:
                    continue
                G, st = W.grad.float(), self.state[W]
                left = W.shape[0] <= W.shape[1]
                if "t" not in st:
                    st["t"] = 0
                if st["t"] % g["gap"] == 0:                      # refresh the projector
                    U, S, Vh = torch.linalg.svd(G, full_matrices=False)
                    st["P"] = U[:, :g["rank"]] if left else Vh[:g["rank"]].T
                P = st["P"]
                R = P.T @ G if left else G @ P                   # r x n  or  m x r
                if "M" not in st:
                    st["M"], st["V"] = torch.zeros_like(R), torch.zeros_like(R)
                st["t"] += 1; t = st["t"]
                st["M"].mul_(b1).add_(R, alpha=1 - b1)
                st["V"].mul_(b2).addcmul_(R, R, value=1 - b2)
                N = (st["M"] / (1 - b1 ** t)) / ((st["V"] / (1 - b2 ** t)).sqrt() + g["eps"])
                upd = P @ N if left else N @ P.T
                W.mul_(1 - g["lr"] * g["wd"]).add_(upd.to(W.dtype), alpha=-g["lr"] * g["scale"])

# galore = [p for n, p in model.named_parameters() if p.ndim == 2 and ("attn" in n or "mlp" in n)]
# rest   = [p for p in model.parameters() if all(p is not q for q in galore)]
# opt = GaLoreAdamW([{"params": galore}], lr=1e-2, rank=256); opt_rest = torch.optim.AdamW(rest, lr=2.5e-3)
# or with Transformers: TrainingArguments(optim="galore_adamw", optim_target_modules=["attn", "mlp"],
#                                         optim_args="rank=256, update_proj_gap=200, scale=0.25")

In POFE, GaLore is offered as an optimiser option on full-parameter recipes, next to AdamW and 8-bit AdamW, with the captured-gradient fraction logged per layer so stale subspaces show up on the dashboard.

Further reading#

Zhao et al. (2024) is the paper; Section 3 has the low-rank gradient analysis and the convergence theorem, Section 4 the memory accounting. Hu et al. (2021) and Lialin et al. (2023, ReLoRA) are the low-rank weight methods it is positioned against. Dettmers et al. (2021) is the 8-bit optimiser that 8-bit GaLore builds on. Zhang et al. (2024, Q-GaLore) quantises the projectors and weights further and adapts the refresh interval per layer.

Frequently asked questions#

References#

1. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. arxiv.org/abs/2403.03507
2. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
3. Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, Anna Rumshisky (2023). ReLoRA: High-Rank Training Through Low-Rank Updates. arxiv.org/abs/2307.05695
4. Tim Dettmers, Mike Lewis, Sam Shleifer, Luke Zettlemoyer (2021). 8-bit Optimizers via Block-wise Quantization. arxiv.org/abs/2110.02861
5. Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, Zhangyang Wang (2024). Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients. arxiv.org/abs/2407.08296
6. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, et al. (2024). LoRA Learns Less and Forgets Less. arxiv.org/abs/2405.09673

Related reading