QLoRA, explained

QLoRA is LoRA on top of a base model whose frozen weights are stored in 4 bits. The pretrained weights are quantised once to a 4-bit data type called NormalFloat (NF4), kept frozen, and dequantised to bf16 block by block on the fly during every forward and backward pass; gradients flow through them into ordinary bf16 LoRA adapters, which are the only parameters that train. The result is fine-tuning at roughly half a byte per base parameter instead of two, which is what lets a 65B model be fine-tuned on a single 48 GB GPU. Like LoRA, QLoRA is a parameterisation and memory strategy, not a loss: it is used under SFT, DPO, ORPO and RL objectives alike, and it sits in the post-training stack wherever the hardware is the binding constraint.

The problem and where it came from#

LoRA removes gradients and optimiser state for the base model but still stores the base in 16 bits: 2 bytes per parameter, so about 140 GB for a 70B model. Quantising weights for inference was well established by 2023 (Dettmers et al.'s LLM.int8() showed 8-bit inference for 175B models without degradation), but training through quantised weights was assumed to cost quality. Dettmers, Pagnoni, Holtzman and Zettlemoyer (2023) showed that it need not. Their paper introduced three pieces: the NF4 data type, double quantisation of the quantisation constants, and paged optimisers to absorb memory spikes. They fine-tuned more than 1,000 models and reported that QLoRA preserved 16-bit fine-tuning task performance; their Guanaco models reached 99.3% of ChatGPT's level on the Vicuna benchmark after 24 hours of fine-tuning on one GPU. The paper also made a finding that changed LoRA practice generally: with LoRA on only the query and value projections they could not match full fine-tuning, but with adapters on all linear layers they could, and the rank then mattered little.

The math: what is stored and what is computed#

Blockwise absmax quantisation

Flatten a weight matrix $W$ and split it into blocks of $B=64$ consecutive values. For block $b$ with values $w_{b,1},\dots ,w_{b,64}$, store one scale $c_b={\max }_j|w_{b,j}|$ and, for each value, the index of the nearest code in a 16-entry codebook $\{q_0,\dots ,q_{15}\}\subset [-1,1]$:

The index takes 4 bits. The blockwise scale is what keeps quantisation local: one outlier only inflates the step size of its own 64-value block instead of the whole tensor.

NormalFloat 4 (NF4)

Pretrained weights are approximately zero-mean Gaussian, and after absmax normalisation every block follows roughly the same distribution. Quantile quantisation is optimal for such data in the sense that each code receives the same expected number of values. Dettmers et al. place the codes at averages of adjacent quantiles of the standard normal,

with $Q_X$ the standard normal quantile function and $k=4$, then renormalise into $[-1,1]$. To guarantee an exact zero (needed for padding and zero-valued weights) they build the negative and positive halves separately, 8 levels on one side and 9 on the other sharing zero, giving 16 codes. Reproducing the construction used in the bitsandbytes library, we get

The codes are dense near zero, where most weights are, and sparse in the tails. A uniform INT4 grid spends as many levels on the rare large values as on the common small ones.

Double quantisation

With fp32 scales and a blocksize of 64, the scales alone cost $32/64=0.5$ bits per parameter. Double quantisation quantises the scales themselves to 8-bit floats in blocks of 256, with one fp32 second-level constant per 256 scales:

a saving of 0.373 bits per parameter, about 3 GB on a 65B model, as reported in the paper.

The forward pass and gradients

For an adapted linear layer with input $X$ the computation is

where $\mathrm{dDQ}$ first dequantises the 8-bit scales with the fp32 second-level constants, then dequantises the NF4 indices with those scales, producing a bf16 matrix $\tilde{W}$ that exists only transiently. In the backward pass the adapter gradients are those of LoRA with $\tilde{W}$ in place of $W_0$: $\partial \mathcal{L}/\partial B=\gamma G{A}^{\top }$ and $\partial \mathcal{L}/\partial A=\gamma B^{\top }G$. The gradient with respect to the input, $\partial \mathcal{L}/\partial X=\delta (\tilde{W}+\gamma BA)$, needs $\tilde{W}$ again, so the block is dequantised a second time. No gradient is ever computed for $W^{\text{NF4}}$ itself. The model being fine-tuned is therefore $\tilde{W}+\gamma BA$, not $W_0+\gamma BA$: the adapter learns on top of the quantised base, and it should be served on top of the same quantised base, or merged with care.

Why training through a quantised base works

Quantisation-aware training usually needs a straight-through estimator, because rounding has zero gradient almost everywhere. QLoRA sidesteps that problem entirely: the quantised weights are constants, never updated, so nothing is differentiated through the rounding. The gradients that matter are with respect to activations (exact, given $\tilde{W}$) and with respect to the adapters (exact, given $\tilde{W}$). The optimisation problem is therefore a clean one: fine-tune the model $\tilde{W}+\gamma BA$, whose starting point differs from the original by quantisation noise of about 0.85% of the weight energy per matrix in our toy. Two facts make that noise tolerable. Fine-tuning moves the function by much more than the noise does, and the noise is close to full rank (see the toy run below) with no particular reason to align with the task gradient, so it acts like a small fixed perturbation of the starting point rather than a bias the adapter must fight. Where it does bite is at very low bit widths, where the starting point is far enough from the original model that the first part of training is spent recovering lost capability.

QLoRA next to its siblings

LoftQ (Li et al., 2023) alternates quantisation and SVD so that $\tilde{W}+BA\approx W_0$ at initialisation, rather than $\tilde{W}$ alone; it matters most at 2 bits. QA-LoRA (Xu et al., 2023) restricts the adapter so the fine-tuned model can be folded back into an INT4 model, which is what you want when the quantised model is also the serving artefact.

Worked example: quantising one block by hand#

Take a block of 8 weights (real blocks have 64, but the arithmetic is identical): $0.031,-0.012,0.004,-0.047,0.020,0.009,-0.026,0.015$. The absmax is $c=0.047$. Dividing gives normalised values $0.660,-0.255,0.085,-1.000,0.426,0.191,-0.553,0.319$. Snapping each to the nearest NF4 code gives $0.7230,-0.2844,0.0796,-1,0.4407,0.1609,-0.5251,0.3379$, stored as 4-bit indices 14, 4, 8, 0, 12, 9, 2, 11. Multiplying back by 0.047 reconstructs $0.0340,-0.0134,0.0037,-0.0470,0.0207,0.0076,-0.0247,0.0159$. The absolute errors are 0.0030, 0.0014, 0.0003, 0, 0.0007, 0.0014, 0.0013 and 0.0009. The largest value is reconstructed exactly (it defines the scale), the small values near zero fall on closely spaced codes, and the worst error is on 0.031, which falls in the wider gap between 0.5626 and 0.7230. We computed these with numpy using the codebook above.

Now memory for a Llama-style 8B model. At 4.127 bits per parameter the quantised linear weights take about $8\times 10^9\times 4.127/8\approx 4.1$ GB. Embedding and output layers are typically left in bf16 (about 1 GB for a 128k vocabulary at 4096 dimensions, each). LoRA at $r=16$ on all linear layers adds about 42M parameters, roughly 0.6 GB including gradients and AdamW state. So the static footprint is around 6 to 7 GB, against about 17 GB for bf16 LoRA and 128 GB for full fine-tuning. At 70B, the quantised base is about 36 GB, which is why QLoRA of a 70B model fits on a single 80 GB GPU with room for activations, and a 65B model fitted on 48 GB in the original paper.

A toy run: how much does NF4 lose, and can LoRA win it back?#

We implemented blockwise absmax quantisation in numpy with three 4-bit codebooks (NF4 as above, a symmetric 15-level INT4 grid, and the E2M1 FP4 format scaled to $[-1,1]$) and applied them to $2^{20}$ synthetic weights with standard deviation 0.02, seed 0. We report relative mean squared error $\mathbb{E}[(\tilde{w}-w{)}^2]/\mathbb{E}[w^2]$.

On Gaussian weights with blocksize 64, NF4's relative error was 0.845%, against 1.117% for FP4 and 1.160% for INT4, a signal-to-quantisation-noise ratio of 20.73 dB versus 19.52 and 19.36 dB. With heavier-tailed Student-t weights (5 degrees of freedom, same variance) the gap widened for INT4 (2.131%) while NF4 rose only to 1.095% and FP4 to 1.261%. Real LLM weights sit between these two cases, with occasional large outliers, which is why both the Gaussian-shaped codebook and the small blocks matter.

Blocksize trades precision for scale storage. NF4 error rose steadily from 0.652% at blocksize 16 to 0.847% at 64 and 1.197% at 4,096. Without double quantisation, blocksize 16 would cost 6 bits per parameter; with it, 4.508. The paper's choice of 64 with double quantisation (4.127 bits) is close to the knee of that curve.

Finally, we asked whether a low-rank adapter can absorb the quantisation error. For a 1024×1024 Gaussian matrix quantised to NF4, the best rank-16 approximation of the residual $W-\tilde{W}$ captured 5.87% of its energy, rank 64 captured 21.0%, and rank 256 captured 62.4%. Quantisation noise is close to full rank, so a typical adapter cannot simply undo it. That is fine for QLoRA's purpose: the adapter is not trying to recover $W_0$, it is learning the task on top of $\tilde{W}$, and the paper's results show the noise rarely matters for fine-tuning quality. It explains why methods like LoftQ, which explicitly fit the adapter to the residual at initialisation, help most at 2 bits, where the residual is much larger.

The algorithm, step by step#

load pretrained weights
for each linear layer except embeddings and lm_head:
    split weight into blocks of 64; c1[b] = absmax(block b)
    idx[b, j] = nearest NF4 code of w[b, j] / c1[b]          # 4-bit
    quantise c1 to FP8 in blocks of 256 with fp32 constants c2  # double quantisation
attach LoRA A, B (bf16) to every quantised linear layer; enable gradient checkpointing
optimizer = paged AdamW (32-bit or 8-bit state) over A, B only

for step in range(total_steps):
    forward: for each layer, W_tilde = dequant(dequant(c2, c1), idx) in bf16; y = x W_tilde^T + gamma x A^T B^T
    loss = objective(outputs)                      # SFT, DPO, ORPO, GRPO ...
    backward: dequantise again for dL/dx; accumulate grads for A, B only
    clip grad norm (the paper used 0.3); optimizer.step()

Data needs are those of the objective. QLoRA's original experiments used instruction datasets from a few thousand examples (OASST1, about 9k, produced the best Guanaco model) up to hundreds of thousands, and found that suitability mattered far more than size: the 9k-example OASST1 set outperformed a 450k-example subsample of FLAN v2 for chatbot performance.

Hyperparameters that matter#

The quantisation settings are nearly always the paper's defaults; the LoRA and optimisation settings follow LoRA practice with a few QLoRA-specific adjustments.

Paged optimisers deserve a sentence on mechanism. They allocate optimiser state in unified memory, so when a long-sequence batch causes an activation spike the driver evicts optimiser pages to CPU RAM and pages them back when the update needs them. They do not reduce steady-state memory; they turn occasional out-of-memory crashes into occasional slow steps. The 8-bit optimiser variant (Dettmers et al., 2021) does reduce steady-state optimiser memory by quantising Adam's moments blockwise, but with adapter-sized state the saving is small.

Compute and memory cost#

where $P_{\text{lin}}$ is the number of parameters in quantised linear layers (0.516 bytes is 4.127 bits), $P_{\text{emb}}$ the embedding and output parameters kept in bf16, and $P_a$ the adapter parameters. Activation memory is unchanged from LoRA and usually dominates at long sequence lengths, so gradient checkpointing is effectively mandatory. Rules of thumb for the static part: about 6 GB at 8B, 10 GB at 14B, 20 GB at 32B, 40 GB at 70B.

The price is speed. Every matmul is preceded by a dequantisation, twice per layer per step (forward and the input-gradient computation in backward), plus a third time if gradient checkpointing recomputes the forward. QLoRA steps are therefore slower than bf16 LoRA steps on the same hardware; the gap depends on kernels, batch size and sequence length, so measure it rather than assume it. If a bf16 LoRA run fits in memory, it is usually the faster and simpler choice. QLoRA is for when it does not fit, or when fitting it would need more GPUs than you have.

Failure modes and how to detect them#

Quality loss at merge time

The adapter was trained against $\tilde{W}$. Merging it into the original bf16 $W_0$ produces $W_0+\gamma BA$, a model nobody trained; merging into $\tilde{W}$ and re-quantising adds a second round of rounding to the update. Symptoms: the merged model scores noticeably worse than the unmerged adapter on the same eval. Remedies: serve the adapter on the same 4-bit base; or dequantise $\tilde{W}$ to bf16, merge, and serve in bf16; or use QA-LoRA if the target is a quantised deployment. Always evaluate the exact artefact you ship.

Loss spikes and NaNs

fp16 compute, a learning rate too high for the model size, or gradient norms above the clip threshold appear as sudden loss spikes and occasionally NaNs. Use bf16 compute, the paper's lower clip value, and log the adapter gradient norm per step; a run that is healthy has a gradient norm that settles within the first few hundred steps.

Silent fp32 casts and frozen norms

Common setup helpers cast layer norms and the output layer to fp32 for stability and enable input gradients so checkpointing works with frozen embeddings. Skipping this step can make the loss refuse to fall (no gradient reaches the adapters) or blow up memory (a bf16 module silently becomes fp32). Check the trainable parameter count and peak memory on step one.

Evaluating in a different precision

A QLoRA model evaluated with a bf16 base looks different from the same model evaluated with its 4-bit base. Pin the precision of the base in your eval harness to whatever you will serve.

When to use QLoRA, and what to use instead#

QLoRA is the right choice when the model does not fit for bf16 LoRA on the hardware you have, when you are iterating on many small experiments on a single GPU, or when the adapter will be served on a 4-bit base anyway. It is the wrong choice when throughput matters more than memory and bf16 LoRA fits, when you need full fine-tuning's capacity, or when the end product is a merged full-precision model and you cannot afford the merge mismatch.

Minimal implementation#

With the open-source Transformers, bitsandbytes and PEFT libraries, QLoRA is a quantisation config plus a LoRA config. In POFE the same choice is a single precision setting on a LoRA recipe.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat 4
    bnb_4bit_use_double_quant=True,      # quantise the scales too
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)

cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)
model.print_trainable_parameters()

opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=2e-4)
# or bitsandbytes.optim.PagedAdamW32bit(...) to page optimiser state on memory spikes
for batch in loader:
    loss = model(**batch).loss                     # SFT; swap in any objective
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 0.3)
    opt.step(); opt.zero_grad(set_to_none=True)

To see what NF4 does to a tensor, the toy above is enough: build the 16-entry codebook, normalise each block of 64 by its absmax, and take the nearest code. The bitsandbytes kernels do exactly that, packed two indices per byte, with the dequantisation fused into the matmul.

Further reading#

Dettmers et al. (2023) is the QLoRA paper; its appendices contain the hyperparameter searches summarised above. Dettmers et al. (2022) on LLM.int8() and Dettmers et al. (2021) on 8-bit optimisers are the precursors. Li et al. (2023, LoftQ) and Xu et al. (2023, QA-LoRA) address the gap between the quantised base and the adapter, from the initialisation side and the deployment side respectively. For the adapter itself, start with the LoRA article.

Frequently asked questions#

References#

1. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arxiv.org/abs/2305.14314
2. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
3. Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arxiv.org/abs/2208.07339
4. Tim Dettmers, Mike Lewis, Sam Shleifer, Luke Zettlemoyer (2021). 8-bit Optimizers via Block-wise Quantization. arxiv.org/abs/2110.02861
5. Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, et al. (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models. arxiv.org/abs/2310.08659
6. Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2309.14717

Related reading