Weight-only quantization: GPTQ, AWQ and keeping your fine-tune

Weight-only quantization stores a model's linear-layer weights in 3 to 8 bits, usually 4, while activations, the KV cache and the arithmetic stay in 16-bit floats. At inference time a fused kernel reads the packed integers, expands them to fp16 or bf16 in registers using a per-group scale and zero point, and multiplies. The format is often written W4A16. GPTQ and AWQ are the two post-training methods that made 4-bit weights accurate enough to ship: both look at a few hundred calibration sequences, both finish in minutes to hours on one GPU, and neither needs gradients through the full model. In the post-training stack they sit at the very end, after SFT, preference tuning or RL and after any LoRA merge, as the last transformation before the weights reach a serving engine. That position is why this article spends as much time on the second half of its title as the first: the quantizer sees your fine-tuned weights, and it has no idea which parts of them are the fine-tune.

The bottleneck: decode reads every weight for every step#

During decode, each step produces one token per sequence and must stream every weight matrix from HBM once (see the inference roofline). For a dense model with $P$ parameters stored at $b$ bits, batch $B$, memory bandwidth $\beta$ and peak matmul throughput $\pi$, a lower bound on step time, ignoring KV-cache traffic, is

The ratio of the two terms is the arithmetic intensity of the weight read, $2PB/(Pb/8)=16B/b$ FLOPs per byte. An H100-class GPU has $\beta \approx 3.35$ TB/s and $\pi \approx 989$ dense bf16 TFLOP/s, a ridge point of about 295 FLOPs per byte. With bf16 weights ($b=16$) decode stays memory-bound until $B\approx 295$; with 4-bit weights plus group metadata ($b\approx 4.25$) the crossover moves down to $B\approx 78$. For an 8B model at batch 1 the bound falls from $16\times 10^9/3.35\times 10^{12}=4.78$ ms to $4.25\times 10^9/3.35\times 10^{12}=1.27$ ms per token, a ceiling of $16/4.25=3.76\times$.

Two things follow. First, weight-only quantization is a latency and capacity technique, not a throughput technique. Its gains are largest for small batches, single-user chat, edge devices and long decodes, and shrink toward zero once batch size pushes the step into the compute-bound regime, because W4A16 kernels still multiply in 16 bits. Frantar et al.'s Marlin kernels (2024) report close to the full 4x for batch sizes up to 16 to 32 and gradually smaller gains up to 64 to 128, which is what the roofline predicts once dequantisation overhead is added. Second, the memory freed is often worth more than the speed: a 70B model drops from about 140 GB to about 37 GB, fits on one 80 GB GPU instead of two, and leaves room for a much larger KV cache, which raises the batch you can run. If you need to go faster at high batch, you need low-precision activations too; that is the subject of FP8 and FP4 inference.

Where GPTQ and AWQ came from#

Round-to-nearest (RTN) quantization with one scale per output channel works at 8 bits and breaks at 4 bits on large language models. Two lines of work fixed it. The first is second-order. Hassibi and Stork's Optimal Brain Surgeon (1992) used the inverse Hessian of the loss to decide which weight to prune and how to adjust the others to compensate. Frantar, Singh and Alistarh's Optimal Brain Compression (2022) made that exact for layer-wise quantization, and Frantar, Ashkboos, Hoefler and Alistarh's GPTQ (2022, ICLR 2023) made it fast: their abstract reports quantizing 175B-parameter models in about four GPU hours to 3 or 4 bits with negligible accuracy loss, and end-to-end speedups over FP16 of about 3.25x on A100 and 4.5x on A6000. The second line is activation-aware scaling. Lin, Tang, Tang, Yang, Chen and colleagues' AWQ (2023, MLSys 2024) observed that protecting about 1% of salient weights greatly reduces error, that salience is better identified from activation magnitudes than from weights, and that it can be protected with an equivalent per-channel scaling instead of mixed precision. AWQ uses no reconstruction or backpropagation, which the authors argue makes it less prone to overfitting the calibration set, a claim our experiment below bears on directly.

The math#

Group-wise asymmetric quantization

Split each row of $W\in {\mathbb{R}}^{d_{\text{out}}\times d_{\text{in}}}$ into groups of $g$ consecutive input channels, typically $g=128$. For a group with values $w_1,\dots ,w_g$ and $b$ bits, store a scale and an integer zero point

and dequantise as ${\hat{w}}_j=s(q_j-z)$. With an fp16 scale and a 4-bit zero per group of 128, storage is $4+16/128+4/128=4.16$ bits per weight; with an fp16 zero it is 4.25. RTN is exactly this formula applied independently to every weight. Its error is the rounding error, at most $s/2$ per weight, with no regard for which weights the next layer actually listens to.

GPTQ: minimise the layer's output error

GPTQ quantises one linear layer at a time. Given calibration inputs $X\in {\mathbb{R}}^{d_{\text{in}}\times n}$ (the layer's inputs on $n$ calibration tokens), it seeks quantised weights that reproduce the layer's outputs:

where $w_i$ is row $i$ and $\mathcal{Q}$ the set of representable values. The objective separates into rows that share one Hessian $H\in {\mathbb{R}}^{d_{\text{in}}\times d_{\text{in}}}$. Optimal Brain Surgeon gives the greedy step for one row: quantise coordinate $q$, then move the not-yet-quantised coordinates $F$ to absorb the error,

after which $q$ is removed from $F$ and the inverse Hessian is updated by one step of Gaussian elimination, $H_{F\setminus q}^{-1}=H_F^{-1}-\frac{1}{[H_F^{-1}{]}_{qq}}(H_F^{-1}{)}_{:,q}(H_F^{-1}{)}_{q,:}$. OBQ chose the coordinate with the smallest $\Delta \mathcal{L}$ per row, which costs $O(d_{\text{out}}{d}_{\text{in}}^3)$. GPTQ's three changes are what make it practical at 100B scale:

A fixed column order for all rows. Quantising columns left to right in the same order for every row loses little accuracy on large layers and means the inverse-Hessian sequence is shared, so the cost drops to $O(d_{\text{in}}^3+d_{\text{out}}{d}_{\text{in}}^2)$. Lazy batched updates. Error compensation is applied inside a block of 128 columns immediately and to the rest of the matrix once per block, turning many rank-1 updates into one matrix multiply. A Cholesky reformulation. The rows of $H^{-1}$ needed by the fixed order are exactly the rows of the upper Cholesky factor of $H^{-1}$, which can be computed once and is numerically stable after adding a dampening term $\lambda I$ with $\lambda =0.01\cdot \mathrm{mean}(\mathrm{diag}H)$.

AWQ: scale salient input channels before rounding

AWQ keeps RTN as the rounding rule and instead changes what gets rounded. For a linear layer $y=Wx$, multiply input channel $j$ of the weight by $s_j$ and divide the corresponding activation by $s_j$, which leaves the full-precision output unchanged: $Wx=(W\mathrm{diag}(s))(\mathrm{diag}(s{)}^{-1}x)$. The division is folded into the preceding RMSNorm or linear layer, so it costs nothing at inference. Quantising the scaled weight gives error on channel $j$ of roughly ${\Delta }^{\prime }\cdot \text{RoundErr}\cdot x_j/s_j$. If $s_j>1$ for a salient channel and the group's step size ${\Delta }^{\prime }$ barely changes (it is set by the group maximum), the effective error on that channel shrinks by a factor of about $s_j$. AWQ searches one exponent per layer group:

where ${\bar{x}}_j$ is the mean absolute activation of input channel $j$ over the calibration set. $\alpha =0$ is plain RTN; $\alpha =1$ scales fully by activation size. The paper uses a grid of 20 values and also searches a weight-clipping ratio. Layers that share an input (q, k and v projections; gate and up projections) share one $s$, because the scale must be folded into a single preceding operator.

Side by side

Worked example: one row, three weights#

Take one row $w=(0.61,-0.36,0.13)$, a grid with step 0.25 and the Hessian

The positive off-diagonal entries say that inputs 1 and 2 are correlated, so an error on weight 1 can be partly cancelled by weight 2. RTN rounds each coordinate to $(0.5,-0.25,0.25)$, with loss $\frac{1}{2}{e}^{\top }He=0.01952$ for $e=w-\hat{w}$. GPTQ goes column by column. Column 1: quantise 0.61 to 0.5, error 0.11, divide by $[H^{-1}{]}_{11}=0.7824$ to get 0.1406, and subtract $0.1406\times (-0.4580,-0.0382)$ from the remaining weights, which moves them to $(-0.2956,0.1354)$. Update the inverse Hessian; its next diagonal entry becomes 0.6098. Column 2: quantise $-0.2956$ to $-0.25$, error $-0.0456$, and the third weight moves by $-0.0274$ to 0.1080. Column 3: quantise 0.1080 to 0. GPTQ's answer is $(0.5,-0.25,0)$ with loss 0.01527, 22% lower than RTN, and a brute-force search over all 729 grid points confirms it is the optimum here. The interesting move is the last weight: on its own 0.13 rounds up, but after absorbing the correlated errors of its neighbours it rounds down. We ran this with numpy; the script is ten lines.

Quantising a fine-tuned model without losing the fine-tune#

Here is the uncomfortable arithmetic. A LoRA or a short SFT changes weights by a small amount relative to their norm, while 4-bit rounding changes them by a lot. In our run below, the merged LoRA delta had Frobenius norm 1.11% of the weights, and INT4 RTN noise was 10.8% of the weights, 9.7 times larger than the fine-tune itself (20.7 times at INT3). Behaviour survives anyway because the noise is roughly isotropic and the fine-tune is a coherent low-rank direction that the network's outputs are sensitive to, but there is no guarantee, and the quantizer's objective decides where the leftover error lands. GPTQ and AWQ minimise output error on the calibration distribution, so if the calibration tokens never exercise the behaviour you fine-tuned for, the quantizer is free to spend its error budget there.

Four practical consequences. Quantise the merged model, not the base, unless you are deliberately serving adapters (see merge or serve the adapter). Calibrate on data that looks like production traffic for the fine-tune: your chat template, your system prompt, your task, ideally including responses written by the fine-tuned model. Evaluate the fine-tune's own metric on held-out data after quantization, because generic perplexity can move in the opposite direction. And treat 3-bit as a different regime, where the method and calibration choice can flip the outcome entirely.

What we measured on a laptop

We ran this end to end on an Apple-silicon Mac (PyTorch 2.10, MPS backend, fp32 fake quantization). The model was Qwen2.5-0.5B-Instruct. We generated greedy answers to 300 templated prompts (60 topics, 5 question types), then built a style fine-tune: targets were the model's own answers lowercased and suffixed with "[done]". A LoRA (rank 16, $\alpha =32$, all seven linear projections in every layer, 8.8M parameters) trained for 60 steps at batch 8, learning rate 1e-4, on 220 prompts, and was merged in fp32. We held out 60 prompts and measured three things: mean log-likelihood per token of the fine-tune's targets, the share of greedy replies (110-token budget) that are entirely lowercase and end in "[done]", and perplexity on held-out English technical prose (about 15k tokens). We implemented RTN, GPTQ (non-sequential: Hessians from the unquantised model in one pass, 128-column blocks, dampening 0.01, no activation reordering) and AWQ (scale search with a 21-point grid, shared scales for layers with a common input, no clipping search), all at group size 128, asymmetric. Embeddings and the tied output head stayed in full precision. Calibration was either 12.3k tokens of generic prose or 12.6k tokens of 140 training conversations in the chat template, including the fine-tuned targets.

The unquantised fine-tune moved target log-likelihood from $-1.687$ to $-0.636$ nats per token and produced the style on 93.3% of held-out prompts (0% before fine-tuning), at the cost of prose perplexity rising from 59.7 to 73.5. Casting the merged model to bf16 changed nothing measurable ($-0.634$, 93.3%). We define retention as the share of the log-likelihood gain that survives,

What the numbers say, with the caveat that this is one seed, one small model and 60 evaluation prompts (a binomial standard error of about 4 to 6 points on the style rate):

Calibration data is a fine-tune-preservation knob, and it trades against generic quality. GPTQ calibrated on the fine-tune's own conversations kept 90% of the log-likelihood gain and the full 93.3% style rate at INT4. Calibrated on generic prose, the same algorithm kept only 72% of the gain, but gave the best prose perplexity of any INT4 run (76.8, against 73.5 unquantised). GPTQ is doing exactly what its objective says: it reconstructs outputs on the distribution it is shown and pays for it elsewhere. Its domain-calibrated INT4 model had the same prose perplexity as RTN.

AWQ is less sensitive to the calibration set. Its searched exponents averaged 0.32 under generic calibration and 0.33 under fine-tune calibration, and its INT4 retention was 81% either way. This matches the paper's argument that fitting one scalar per layer group cannot overfit the way re-rounding every weight can. At INT3, AWQ was the only method that kept the style reasonably intact with generic calibration (71.7% style rate, where GPTQ collapsed to 0%).

Log-likelihood and behaviour disagree at the margins. AWQ with generic calibration retained more log-likelihood than GPTQ with generic calibration (81% vs 72%) but produced the style less often (71.7% vs 85%): the "[done]" marker at the end of long replies is a single token whose probability can drop below the threshold for greedy decoding without moving the average much. Measure the behaviour you shipped, not a proxy.

Keeping the adapter separate did not protect it. Quantising the base and adding the exact fp32 adapter on top kept 72.5% (RTN) and 64.3% (GPTQ) of the gain, no better than quantising the merged model, and much worse than domain-calibrated GPTQ on the merged weights. The adapter was trained on the unquantised base; moving the base under it changes the function the adapter was correcting. This is the same mismatch that motivates QLoRA (train on the quantised base you will serve) and QA-LoRA (train adapters that fold into the quantised grid).

INT3 is a cliff for small models. At three bits, every configuration lost most of the fine-tune and prose perplexity rose to 105 to 378. Larger models degrade far more gracefully (the GPTQ paper's 3-bit results are on 175B models), but on any model, 3-bit deserves a fresh evaluation rather than an extrapolation from 4-bit.

The procedure, step by step#

# inputs: fine-tuned model (LoRA already merged, bf16), calibration set C
for layer in model.layers:                     # sequential: feed quantised outputs forward
    X = inputs_to(layer, C)                    # 128 to 512 sequences of 512 to 2048 tokens
    for group in [(q,k,v), (o,), (gate,up), (down,)]:
        if method == "awq":
            xbar = mean(abs(X[group]), over tokens)
            alpha = argmin over grid(0..1, 20 steps) of
                    sum ||RTN(W*xbar**alpha) @ (X/xbar**alpha) - W @ X||^2
            fold 1/xbar**alpha into the previous norm or linear; W *= xbar**alpha
            Q = RTN(W, bits=4, group=128)
        if method == "gptq":
            H = 2 X X^T + 0.01*mean(diag) I;  U = chol(inverse(H), upper)
            for block of 128 columns: for column c in block:
                q = quant(W[:, c]); e = (W[:, c] - q) / U[c, c]
                W[:, c+1:block_end] -= e outer U[c, c+1:block_end]
              W[:, block_end:] -= E_block @ U[block, block_end:]
    X_next = layer_with_Q(X)                   # GPTQ in practice is sequential like this
pack ints two per byte (or 8 per int32), save scales and zeros, keep embeddings and lm_head in 16-bit
evaluate the exact artefact you will serve on the fine-tune's metric

Calibration size matters less than calibration content. The original papers used 128 random sequences of 2,048 tokens from generic web text; open-source tooling now commonly defaults to a few hundred chat samples. For a fine-tuned model, a reasonable starting set is 256 to 512 examples from the fine-tune's training or validation data, rendered with the exact chat template and system prompt used in production, mixed with 10 to 30% generic text if general ability also matters. Our experiment suggests the mix is worth tuning: pure fine-tune calibration protected the fine-tune and cost general perplexity.

Tuning knobs#

Interaction with post-trained artefacts#

LoRA adapters. Merge first, then quantise, if you serve one model. If you serve many adapters on one base (multi-LoRA serving), the base is quantised once without any adapter in it, so evaluate every adapter against the quantised base, and consider training adapters on that quantised base so there is no mismatch. Custom chat templates and special tokens. If calibration text never contains your template's control tokens, the activations they produce are absent from $H$, and the layers are not fitted for them. Our domain calibration included the template; it is the cheapest insurance there is. Distilled or reasoning models. Long chains of thought amplify small per-token errors, since every token conditions the next; evaluate with full-length generations, not teacher-forced likelihood. MoE models. Each expert sees only the tokens routed to it, so rare experts get Hessians estimated from very few tokens; increase calibration size or dampening for them. Merged models from model merging behave like any other dense checkpoint, but calibrate on all the tasks the merge is meant to preserve.

When weight-only quantization helps, and when to use something else#

Failure modes and how to detect them#

The fine-tune quietly fades. Generic benchmarks look fine and the behaviour you paid for is weaker. Detect with the fine-tune's own held-out eval, before and after, and track retention as defined above. Fix with calibration data in production format. Perplexity improves while the task degrades, or the reverse; our GPTQ runs show both directions. Never sign off on perplexity alone. One layer explodes. Per-layer relative output error $\Vert WX-\hat{W}X\Vert /\Vert WX\Vert$ above a few percent at INT4 is a warning; log it during quantization. Common causes are activation outliers, singular Hessians from dead inputs, and too little calibration data for a rare expert. Long-context drift. Calibration at 512 tokens does not exercise positions at 32k; test long prompts. Kernel mismatch. A checkpoint with activation ordering, an unusual group size or symmetric zero points may fall back to a slow kernel, erasing the speedup; check that the engine reports the fast path and measure tokens per second. No speedup at high batch is not a bug, it is the roofline.

Implementation sketch#

The open-source llm-compressor library (from the vLLM project) produces checkpoints that vLLM loads directly; the API below follows its current examples and changes between versions, so check the repository. GPTQModel and the older, now archived AutoAWQ are alternatives, and SGLang and TensorRT-LLM serve the same formats. In POFE this is a quantization step at the end of a recipe, calibrated on the recipe's own validation split.

from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import Dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model = AutoModelForCausalLM.from_pretrained(MERGED_FINETUNE_DIR, torch_dtype="auto")
tok = AutoTokenizer.from_pretrained(MERGED_FINETUNE_DIR)

# calibration = the fine-tune's own conversations, in the production chat template
texts = [tok.apply_chat_template(conv, tokenize=False) for conv in val_conversations[:512]]
calib = Dataset.from_dict({"text": texts})

recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
oneshot(model=model, dataset=calib, recipe=recipe,
        max_seq_length=2048, num_calibration_samples=len(texts))
model.save_pretrained(OUT_DIR, save_compressed=True); tok.save_pretrained(OUT_DIR)
# serve: vllm serve OUT_DIR   (quantization format is read from the checkpoint config)

The core of GPTQ itself is short. This is the inner loop we used for the experiment above, in PyTorch:

def gptq(W, H, bits=4, gs=128, damp=0.01, bs=128):
    W = W.clone().double(); H = H.clone().double(); c = W.shape[1]
    H += damp * H.diag().mean() * torch.eye(c, dtype=H.dtype)
    U = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True)
    Q = torch.zeros_like(W)
    for i1 in range(0, c, bs):
        i2 = min(i1 + bs, c); W1 = W[:, i1:i2].clone(); E1 = torch.zeros_like(W1)
        for i in range(i2 - i1):
            if (i1 + i) % gs == 0: s, z, qmax = group_params(W1[:, i:i + gs], bits)
            q = dequant(quant(W1[:, i], s, z, qmax), s, z)
            e = (W1[:, i] - q) / U[i1 + i, i1 + i]
            W1[:, i:] -= e[:, None] * U[i1 + i, i1 + i:i2][None, :]
            Q[:, i1 + i] = q; E1[:, i] = e
        W[:, i2:] -= E1 @ U[i1:i2, i2:]
    return Q.float()

Further reading#

Frantar et al. (2022) for GPTQ and Frantar, Singh and Alistarh (2022) for the OBS-based framework underneath it; Hassibi and Stork (1992) for the original Optimal Brain Surgeon. Lin et al. (2023) for AWQ. Frantar et al. (2024) on Marlin explains why weight-only speedups survive moderate batch sizes and where they stop. Dettmers et al. (2022) on LLM.int8() is the 8-bit precursor and the first careful description of activation outliers. For the fine-tuning side of the mismatch, see QLoRA, Xu et al. (2023) on QA-LoRA and Li et al. (2023) on LoftQ. For activations as well as weights, continue to FP8 and FP4 inference.

Frequently asked questions#

References#

1. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arxiv.org/abs/2210.17323
2. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arxiv.org/abs/2306.00978
3. Elias Frantar, Sidak Pal Singh, Dan Alistarh (2022). Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. arxiv.org/abs/2208.11580
4. Babak Hassibi, David G. Stork (1992). Second order derivatives for network pruning: Optimal Brain Surgeon. proceedings.neurips.cc/paper/1992/hash/303ed4c69846ab36c2904d3ba8573050-Abstract.html
5. Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler, Dan Alistarh (2024). MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models. arxiv.org/abs/2408.11743
6. Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arxiv.org/abs/2208.07339
7. Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2309.14717
8. Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, et al. (2023). LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models. arxiv.org/abs/2310.08659
9. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arxiv.org/abs/2404.00456
10. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, et al. (2024). SpinQuant: LLM quantization with learned rotations. arxiv.org/abs/2405.16406
11. vLLM project (2024). llm-compressor (open-source library). github.com/vllm-project/llm-compressor

Related reading