FP8 and FP4 inference, SmoothQuant and activation quantization

FP8 and FP4 inference quantize both operands of every large matrix multiply, the weights and the activations, so that the tensor cores themselves run at 8 or 4 bits. That is the difference from weight-only quantization: W4A16 shrinks the bytes read from memory but still multiplies in bf16, while W8A8 and W4A4 also raise the arithmetic ceiling, which is what matters for prefill, long prompts and large-batch decode. The hard part is activations. Weights are fixed and well behaved; activations change with every token and, in transformers, carry a few channels whose magnitudes are hundreds or thousands of times the median. SmoothQuant, per-token scales, floating-point formats with wide dynamic range and small-block scaling (MXFP4, NVFP4) are four answers to that one problem. In the post-training stack this is a serving-time transformation applied after fine-tuning, and increasingly also inside RL loops, where rollouts are sampled from a low-precision copy of the policy being trained.

The bottleneck: compute-bound phases do not care about weight bytes#

A linear layer with $P$ weights processing $B$ tokens performs $2PB$ FLOPs and reads at least $P{b}_w/8$ bytes of weights. With memory bandwidth $\beta$ and peak throughput $\pi (b_a,b_w)$, which depends on the precision of both operands, the step time is bounded below by

Weight-only INT4 shrinks the first term and leaves the second unchanged. FP8 W8A8 halves the first term (8 bits instead of 16) and halves the second too, because FP8 tensor cores on Hopper-class GPUs deliver twice the dense bf16 rate (about 1,979 versus 989 TFLOP/s on an H100). FP4 on Blackwell-class GPUs doubles the rate again over FP8. Prefill is compute-bound almost from the first token, since a 2,000-token prompt is $B=2000$ for the weight read, so time to first token improves only with low-precision math. The same holds for decode once the batch passes the ridge point (see the inference roofline and continuous batching).

For an 8B model the ideal bound at batch 1 is 4.78 ms per step in bf16, 1.27 ms with INT4 weights and 2.39 ms with FP8. At batch 512 the order flips: bf16 and W4A16 both need 8.28 ms, FP8 needs 4.14 ms. Real engines get less than the ideal (SmoothQuant's paper reports up to 1.56x end-to-end for INT8 W8A8; QServe reports 20 to 90% runtime overhead for naive INT4 dequantization in large-batch serving), but the shape holds: weight-only for latency at small batch, W8A8 for throughput, and combinations such as W4A8 that try to get both.

Where it came from#

INT8 inference was standard for convolutional networks well before LLMs. It broke on large transformers because of activation outliers: Dettmers, Lewis, Belkada and Zettlemoyer (LLM.int8(), 2022) described highly systematic emergent outlier features in a small number of hidden dimensions, and handled them by routing those dimensions through a 16-bit matmul, which preserved accuracy but was slow. Yao et al.'s ZeroQuant (2022) used fine-grained scales, group-wise for weights and per-token for activations. Xiao, Lin, Seznec, Wu, Demouth and Han's SmoothQuant (2022, ICML 2023) moved the difficulty from activations to weights with an exact rescaling, making plain INT8 W8A8 kernels accurate for models up to 530B. In parallel, Micikevicius et al. (2022) proposed the two FP8 encodings, E4M3 and E5M2, that Hopper GPUs implemented, and noted that FP8 post-training quantization worked on language models that had resisted INT8. Van Baalen et al. (2023) argued the opposite trade-off for edge hardware, where INT8 logic is cheaper. For 4 bits, Rouhani et al. (2023) evaluated the OCP Microscaling (MX) formats, which pair narrow elements with a shared power-of-two scale per block of 32, and NVIDIA's NVFP4 (described in their 2025 NVFP4 pretraining paper) uses blocks of 16 with an FP8 scale. Rotation methods, QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2024), remove outliers by multiplying activations and weights by orthogonal matrices before quantizing.

The number formats#

A floating-point value with $e$ exponent bits and $m$ mantissa bits is $(-1{)}^S{2}^{E-\text{bias}}(1+M/2^m)$, with subnormals below the smallest exponent. Within each power-of-two interval the spacing is uniform with $2^m$ steps, so the relative rounding error is bounded by $2^{-(m+1)}$ everywhere in the normal range: at most 6.25% for E4M3, 12.5% for E5M2 and 25% for E2M1. INT8 instead has a uniform step $s=\max |x|/127$, so its relative error on a value $x$ is about $s/(2|x|)$: tiny for the largest values and unbounded for values much smaller than the scale. That is the whole INT-versus-FP debate in one line. If the tensor is roughly uniform within its range, INT8's 256 evenly spaced levels beat FP8. If a tensor mixes magnitudes over several orders, as LLM activations do, FP8's logarithmic spacing keeps small values representable while a single per-tensor INT8 scale flushes them to zero.

Scale granularity

Every format needs a scale $s$ mapping real values into its range, $\hat{x}=s\cdot {\mathrm{round}}_{\mathcal{F}}(x/s)$. Choices, from coarse to fine: per-tensor (one scale, either fixed from calibration, "static", or computed from the current tensor, "dynamic"); per-channel for weights (one scale per output row); per-token for activations (one scale per row of $X$, computed on the fly); and per-block, such as 1×128 activation tiles and 128×128 weight blocks in DeepSeek-V3's FP8 recipe, or 16- and 32-element blocks in NVFP4 and MXFP4. Per-channel weight scales and per-token activation scales are the finest granularity that still factors out of the matmul, since $\hat{X}{\hat{W}}^{\top }=\mathrm{diag}(s_X)(X_q{W}_q^{\top })\mathrm{diag}(s_W)$ can be applied as an epilogue after an integer or FP8 GEMM. Scales per input channel do not factor out this way, which is why activation outliers, which live in specific input channels, cannot be fixed by finer scales alone without block-scaled hardware.

SmoothQuant: move the outliers into the weights#

For $Y=X{W}^{\top }$ with $X\in {\mathbb{R}}^{n\times d}$, any positive vector $s\in {\mathbb{R}}^d$ gives an exact identity

Dividing activation channel $j$ by $s_j$ shrinks the outlier channels; multiplying weight column $j$ by $s_j$ makes those weights larger, and weights have plenty of headroom. SmoothQuant chooses

so that after scaling, $\max |{\hat{X}}_{:j}|=\max |{\hat{W}}_{:j}|=(\max |X_{:j}|\cdot \max |W_{:j}|{)}^{1/2}$ when $\alpha =0.5$: the per-channel ranges of activations and weights become equal. Larger $\alpha$ pushes more difficulty onto the weights; the paper uses 0.5 for most models and up to about 0.75 to 0.85 for some with severe outliers. The division by $s$ is folded offline into the preceding operator (an RMSNorm or LayerNorm gain, or the previous linear layer's output rows), so inference runs a plain W8A8 GEMM. The maxima come from a few hundred calibration sequences. This is the same equivalence that AWQ uses, applied for the opposite reason: AWQ scales to protect salient weights, SmoothQuant to tame activations.

Worked example: three channels, one outlier

Two tokens, three input channels, two outputs, and 4-bit symmetric per-tensor quantization (levels $-7$ to $7$) so the effect is visible by hand:

Channel 3 is an outlier with tiny weights. The activation step is $58/7=8.29$, so channels 1 and 2 round to zero, the weight step is $0.55/7=0.079$, so channel 3's weights round to zero, and the quantized product is exactly zero: 100% error. Now smooth with $\alpha =0.5$. The channel maxima are $\max |X_{:j}|=(1.3,0.9,58)$ and $\max |W_{:j}|=(0.42,0.55,0.020)$, so $s=(1.759,1.279,53.85)$. After scaling both steps are about 0.154, the quantized activations are $(0.769,-0.308,1.077)$ and $(-0.462,0.769,-0.769)$, and the quantized product is $\left(\begin{array}{cc}1.397& -1.633\\ -1.184& 1.562\end{array}\right)$, a relative L1 error of 5.3%. Same bits, same kernel, one offline rescaling. We computed this with numpy.

Measured: a 0.5B instruct model on a laptop#

We fake-quantized Qwen2.5-0.5B-Instruct in fp32 on an Apple-silicon Mac (PyTorch 2.10, MPS), emulating each format in software: E4M3 and E5M2 with correct subnormals and saturation, E2M1 for FP4, MXFP4 with a power-of-two scale per 32 values, NVFP4 with an E4M3 scale per 16 values and an fp32 tensor scale. All seven linear projections in all 24 layers were quantized; attention score and value matmuls, the KV cache, embeddings and the tied output head stayed in fp32. We compared each configuration with the unquantized model on 60 held-out chat prompts paired with the model's own greedy answers (about 3,800 response tokens), reporting mean per-token KL divergence $\mathrm{KL}(p_{\text{fp32}}\Vert p_{\text{quant}})$, the share of positions where the argmax token agrees, and perplexity on about 15k tokens of held-out English technical prose (fp32 baseline 59.70). SmoothQuant maxima came from 48 sequences of 256 prose tokens.

First, the outliers. Over 24 layers, the ratio of the largest to the median per-channel max $|x|$ was typically 10 to 12 for the inputs to q/k/v and to gate/up, and 3.5 for the o_proj input. The MLP down-projection input was the outlier site: at layer 2 one channel reached 2,027 against a median of 1.65, a ratio of 1,232; the next largest channel was 37.5.

Five observations. Per-tensor INT8 activations fail on this model, and SmoothQuant applied only where the paper folds it (the norm-fed q/k/v and gate/up inputs) did not rescue it, because the catastrophic outlier sits in the down-projection input. Smoothing that site too (folding $1/s$ into the up-projection's output rows, which is exact because the down input is $\mathrm{SiLU}(\text{gate})\odot \text{up}$) cut KL from 1.49 to 0.52, still unusable; the $\alpha$ sweep was erratic (0.25 gave 2.02, 0.75 gave 1.71, 0.9 gave 2.84), a sign that one per-tensor scale cannot span a 1,232x range in 8 uniform bits however it is shared. Per-token activation scales are the real fix for INT8: KL 0.027, and SmoothQuant on top improved it further to 0.018. FP8 E4M3 needs neither trick: even with one scale per tensor it matched smoothed per-token INT8 (0.0195 vs 0.0184), because its logarithmic spacing keeps small values alive under a large scale. This is why FP8 displaced INT8 W8A8 on GPUs that support it. E5M2 is the wrong FP8 for inference: its extra exponent bit buys range that a scaled tensor does not need and costs a mantissa bit, quintupling KL. At 4 bits, the scale format matters as much as the element format: NVFP4's 16-element blocks with E4M3 scales had less than a third of MXFP4's KL at W4A4 (0.25 vs 0.84), and most of NVFP4's error came from the weights (W4A16 alone was 0.131). A 0.5B model is a harsh test; small models tolerate quantization noise worse than large ones, so treat these numbers as a ranking, not a forecast for 70B.

The pipeline, step by step#

# inputs: fine-tuned bf16 model, ~256-512 calibration sequences in production format
1. choose a scheme per layer type:
     FP8 W8A8:   weights per-channel (static), activations per-token (dynamic)  # default on FP8 GPUs
     INT8 W8A8:  SmoothQuant scales first, then per-channel / per-token
     NVFP4/MXFP4: block scales; often W4A16 or W4A8 first, W4A4 only after evaluation
2. skip sensitive layers: lm_head, embeddings, MoE router; optionally first/last block
3. (INT8 or 4-bit) run calibration: collect per-channel max|X| for smoothing,
     static activation scales if not dynamic, GPTQ Hessians if weights use GPTQ
4. fold smoothing scales into the previous RMSNorm gain or linear output rows
5. quantize weights, store scales; set KV-cache dtype (FP8 with per-tensor or per-head scales)
6. evaluate the exact artefact: task metrics, KL vs bf16 on production prompts, long-context tests

FP8 with dynamic per-token activation scales needs no calibration at all, which is why many engines can apply it at load time. Static activation scales save a reduction per token but need calibration data that covers the production range; if live traffic exceeds the calibrated maximum, values saturate.

Interaction with post-trained models#

Fine-tunes move outliers. Activation outlier channels are a property of the trained network, and fine-tuning can grow, shrink or move them. Smoothing factors and static scales calibrated on the base model are wrong for the fine-tune; recompute them on the merged fine-tuned weights with fine-tune-style data, exactly as for GPTQ calibration. LoRA adapters on an FP8 base. Serving engines that support multiple LoRA adapters typically run the base GEMM in FP8 and the adapter's two thin matmuls in bf16. The adapter was trained against a bf16 base, so the same base-mismatch caveat as in merge versus adapter applies, though FP8's error is about an order of magnitude smaller than INT4's. RL with low-precision rollouts. When GRPO or PPO samples rollouts from an FP8 copy of the policy and computes gradients in bf16, the sampler and the learner are different distributions, and the importance ratio silently includes quantization error. Our measured FP8 KL of about 0.02 nats per token sounds small, but over a 1,000-token rollout it is a sequence-level log-ratio of order 20 nats, which is why such pipelines track sampler-learner KL and apply importance-ratio corrections or clipping. Custom tokenizers. Newly added embedding rows are not quantized (embeddings stay 16-bit), but the first time the model sees them in an activation they can be outliers; include them in calibration. Distilled students trained with knowledge distillation are no more robust than other models of their size; evaluate them on their own.

Tuning knobs#

When it helps, when it hurts#

Failure modes and how to detect them#

Static scales saturate on unseen inputs. Long prompts, unusual languages or tool outputs push activations past the calibrated maximum and get clipped; quality drops on exactly the traffic that differs from calibration. Detect by logging the fraction of saturated values per layer on live traffic, or use dynamic scales. One layer carries all the error. Compute per-layer KL by quantizing one layer at a time; in our model the down-projection input at early layers is the place to look. Format confusion. E5M2 for forward activations, or MXFP4 where NVFP4 was intended, quietly costs several times the error, as our table shows. Underflow of small activations under INT8 per-tensor scaling looks like a sudden collapse of top-1 agreement (to 49% in our run) rather than gradual drift. Sampler-learner drift in RL appears as growing importance-ratio variance or clip fractions while reward stalls. A KV cache in FP8 without proper scales degrades long-context retrieval while short benchmarks look fine. In every case the cheapest monitor is the same: KL against the bf16 model on a few hundred production prompts, broken down by layer and by position.

Implementation sketch#

vLLM can apply FP8 at load time with dynamic scales (--quantization fp8) and store the KV cache in FP8 (--kv-cache-dtype fp8); SGLang and TensorRT-LLM offer equivalent options. For a calibrated checkpoint, the open-source llm-compressor library writes formats those engines read. The scheme names below follow its current examples and may change between versions. In POFE, the serving precision is part of the evaluation, so the eval runs against the quantized artefact rather than the bf16 checkpoint.

from transformers import AutoModelForCausalLM
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained(MERGED_FINETUNE_DIR, torch_dtype="auto")
# FP8: per-channel weights, dynamic per-token activations, no calibration data needed
recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
oneshot(model=model, recipe=recipe)
model.save_pretrained(OUT_DIR, save_compressed=True)
# INT8 W8A8 instead: put a SmoothQuantModifier(smoothing_strength=0.5) before a GPTQ or
# QuantizationModifier with scheme="W8A8", and pass calibration data in production format.

The emulation we used for the measurements, useful for checking a format on your own model before you have the hardware:

def fp_round(x, mbits, emin, maxv):
    e = torch.floor(torch.log2(x.abs().clamp(min=1e-30))).clamp(min=emin)
    step = torch.exp2(e - mbits)                  # spacing within the octave
    return (torch.round(x / step) * step).clamp(-maxv, maxv)
e4m3 = lambda x: fp_round(x, 3, -6, 448.0)
e2m1 = lambda x: fp_round(x, 1, 0, 6.0)

def fp8_per_token(x):                             # dynamic activation scale per row
    s = x.abs().amax(-1, keepdim=True).clamp(min=1e-8) / 448.0
    return e4m3(x / s) * s

def nvfp4(x, block=16):
    xb = x.reshape(-1, block); g = x.abs().amax() / (448 * 6)    # fp32 tensor scale
    s = e4m3(xb.abs().amax(1, keepdim=True) / 6 / g) * g         # E4M3 block scale
    s = torch.where(s == 0, torch.ones_like(s), s)
    return (e2m1(xb / s) * s).reshape(x.shape)

def smooth(W, act_max, alpha=0.5):                # returns s; use X/s and W*s
    return act_max.clamp(min=1e-5)**alpha / W.abs().amax(0).clamp(min=1e-5)**(1 - alpha)

Further reading#

Micikevicius et al. (2022) define E4M3 and E5M2. Xiao et al. (2022) is SmoothQuant; Dettmers et al. (2022) is the LLM.int8() paper that characterised activation outliers; Yao et al. (2022, ZeroQuant) introduced per-token activation and group-wise weight scales for LLMs. Rouhani et al. (2023) and the OCP MX specification cover MXFP4 and its siblings; NVIDIA's NVFP4 pretraining paper (2025) describes NVFP4's two-level scaling. Van Baalen et al. (2023) make the case for INT8 on dedicated inference hardware, and Shen et al. (2023) study FP8 post-training quantization across many models. QuaRot and SpinQuant use rotations to reach 4-bit activations, QServe (Lin et al., 2024) is the reference for W4A8 with a 4-bit KV cache, and the DeepSeek-V3 report documents fine-grained block-scaled FP8 at scale. For the weight-only side, see GPTQ and AWQ.

Frequently asked questions#

References#

1. Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, et al. (2022). FP8 Formats for Deep Learning. arxiv.org/abs/2209.05433
2. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, Song Han (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arxiv.org/abs/2211.10438
3. Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arxiv.org/abs/2208.07339
4. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. arxiv.org/abs/2206.01861
5. Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, et al. (2023). Microscaling Data Formats for Deep Learning. arxiv.org/abs/2310.10537
6. Open Compute Project (2023). OCP Microscaling Formats (MX) Specification v1.0. www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
7. NVIDIA, Felix Abecassis, Anjulie Agrusa, Dong Ahn, et al. (2025). Pretraining Large Language Models with NVFP4. arxiv.org/abs/2509.25149
8. Mart van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, et al. (2023). FP8 versus INT8 for efficient deep learning inference. arxiv.org/abs/2303.17951
9. Haihao Shen, Naveen Mellempudi, Xin He, Qun Gao, Chang Wang, Mengni Wang (2023). Efficient Post-training Quantization with FP8 Formats. arxiv.org/abs/2309.14592
10. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, et al. (2024). QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arxiv.org/abs/2404.00456
11. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, et al. (2024). SpinQuant: LLM quantization with learned rotations. arxiv.org/abs/2405.16406
12. Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, et al. (2024). QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. arxiv.org/abs/2405.04532
13. DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arxiv.org/abs/2412.19437

Related reading