KV cache quantization and eviction: FP8, INT4, H2O and attention sinks

KV cache compression shrinks the per-sequence memory that dominates long-context serving, in two independent ways. Quantisation stores each cached key and value in fewer bits: FP8 halves the cache with little effort, and 4-bit or 2-bit schemes such as KIVI and KVQuant go further by quantising keys and values along different axes. Eviction stores fewer entries: StreamingLLM keeps a handful of initial "attention sink" tokens plus a recent window, H2O keeps the tokens that have received the most attention plus a recent window, and later methods such as SnapKV choose what to keep from the prompt itself. Both attack the same two terms from the roofline: cache capacity, which caps the batch, and cache bandwidth, which sets time per output token at long context. Quantisation is close to lossless at 8 bits and usable at 4 with the right grouping; eviction changes what the model can see and must be judged on the tasks you serve, not on perplexity. This article gives the math for both, runs them on a real post-trained model, and explains where each breaks.

The bottleneck in numbers#

From the KV cache article, each token costs $\kappa =2L{n}_{kv}{d}_{h}b$ bytes, and a decode step reads the cache of every token in the batch. Write the bits per stored element, including quantisation metadata, as $\bar{b}$, and the number of cached entries per sequence as $S_{\text{kept}}\le S$. Then

Quantisation lowers $\bar{b}$; eviction lowers $S_{\text{kept}}$; grouped-query and latent attention lower $n_{kv}{d}_h$ but are fixed when the model is trained. For Llama 3 8B a 128k-token sequence needs 17.2 GB in BF16. FP8 brings it to 8.6 GB. INT4 with one 16-bit scale and one 16-bit zero point per group of 32 elements costs $\bar{b}=4+32/32=5$ bits, 5.4 GB, a 3.2× reduction. Two-bit values with the same metadata cost 3 bits, 3.2 GB. Keeping 20% of tokens, the budget H2O evaluates, turns a 32k-token sequence's 4.3 GB into 0.86 GB; keeping a fixed 1,024 entries, as a StreamingLLM-style policy can, holds the cache at 134 MB however long the stream runs, and makes TPOT independent of sequence length.

Quantising the cache#

The quantiser and why the grouping axis matters

Uniform asymmetric quantisation of a group of values $x_1,\dots ,x_g$ to $b$ bits uses a scale and zero point

The error bound is proportional to the range of the group, so what matters is which values share a group. Cached keys have a few channels whose magnitude is consistently far larger than the rest across tokens. If a group is one token's key vector (per-token grouping), its range is set by the outlier channel and every other channel is rounded with a step far coarser than its own spread. If a group is one channel across a run of tokens (per-channel grouping), the outlier channel gets its own scale. Liu et al. (2024) made this the core of KIVI: quantise keys per-channel and values per-token. Values are grouped per token because the attention output is a weighted sum of whole value vectors, so errors in a token's value vector affect the output in proportion to that token's weight, and value vectors do not show the same fixed outlier channels.

Errors in keys are also amplified in a way errors in values are not. A key error $\Delta k$ shifts the attention logit by $\delta =q^{\top }\Delta k/\sqrt{d_h}$, which multiplies that token's unnormalised weight by $e^{\delta }$. Large query components on the same outlier channels make $\delta$ large exactly where per-token key quantisation is coarsest.

FP8, KIVI and KVQuant

FP8 (E4M3, 4 exponent and 3 mantissa bits, or E5M2) is the production default for halving the cache. It needs a scale per tensor or per head so that values fit the format's range; E4M3's largest finite value is 448. vLLM exposes it as a cache dtype, with scales either set to 1.0 or calibrated on a dataset, optionally per attention head. Its documentation notes that with the FlashAttention-3 backend the attention computation itself also runs in FP8, so queries are quantised too.

KIVI (Liu, Yuan, Jin, Zhong, Xu, Braverman, Chen and Hu, 2024) is a tuning-free 2-bit scheme: keys per-channel and values per-token in groups of 32, with the most recent $R\le 128$ tokens kept in full precision in a residual window until they fill a group. It reports nearly unchanged quality on Llama, Falcon and Mistral models with 2.6× lower peak memory including weights, up to 4× larger batches and 2.35 to 3.47× throughput on real workloads.

KVQuant (Hooper, Kim, Mohammadzadeh, Mahoney, Shao, Keutzer and Gholami, 2024) adds three ideas to per-channel keys: quantise keys before the rotary embedding, which mixes channel pairs and smears outliers; use non-uniform, sensitivity-weighted per-layer datatypes; and store a small fraction of outliers separately in a sparse format per vector. It reports under 0.1 perplexity degradation at 3 bits on Wikitext-2 and C4, LLaMA-7B with up to 1 million tokens of context on one A100-80GB and 10 million on eight GPUs, and custom kernels up to about 1.7× faster than the FP16 baseline for the relevant matrix-vector products.

Toy run: quantising a real model's cache

We implemented Qwen2.5-0.5B-Instruct (24 layers, 14 query heads, 2 key-value heads, head dimension 64, rotary positions) directly in PyTorch from its released weights, checked it against the reference implementation (maximum logit difference $2.6\times 10^{-4}$, identical argmax on all 256 test positions), and measured perplexity on two 2,048-token passages of Pride and Prejudice with every cached key (after rotary embedding) and value fake-quantised. Asymmetric min-max quantisation, per-token groups of 64 (one head vector), per-channel groups of 32 tokens, FP32 compute, no full-precision residual window.

Three things stand out. First, grouping dominates bit width: 4-bit keys grouped per channel beat 8-bit keys grouped per token, and per-token 4-bit collapsed entirely. The reason is visible in the cache: in layer 0 the largest per-channel mean absolute key value was 22 to 25 times the median channel, and 5 to 6 times in layers 12 and 23. Second, FP8 cost this small model 5.5 to 5.9% perplexity, much more than the near-zero loss usually reported for larger models; small models with large key outliers are a worst case, which is exactly why the format should be checked per model. Third, our first FP8 run with calibrated scales returned NaN: values in the second passage exceeded the range calibrated on the first, and casting a value above 448 to E4M3 produces NaN rather than saturating. Clamping before the cast fixed it. Production kernels saturate, but any custom quantisation path should test this. The 2-bit KIVI-style row is far worse than KIVI's reported results because we kept no full-precision residual window and quantised every token, including the most recent ones that receive the most attention.

Evicting tokens#

Eviction keeps a fixed budget of $B_k$ cache entries per head per layer and drops the rest. The question is which entries to keep.

Attention sinks and StreamingLLM

The obvious policy, a sliding window over the most recent tokens, fails badly once the first tokens are evicted. Xiao, Tian, Chen, Han and Lewis (2023) traced this to attention sinks: trained models put a surprisingly large share of attention on the first few tokens regardless of their content. Softmax weights must sum to 1, so a head that has nothing useful to attend to still has to put its mass somewhere, and the initial tokens, visible to every later position during training, become the dumping ground. Evicting them removes a large part of the softmax denominator and shifts every remaining weight. StreamingLLM keeps the key-value states of the first few tokens (four suffice in their experiments) plus a window of recent tokens, and assigns positions by place in the cache rather than in the original text, caching keys before the rotary transformation so they can be re-rotated. With this, Llama-2, MPT, Falcon and Pythia modelled text stably over up to 4 million tokens, up to 22.2× faster than a sliding window that recomputes the cache. The authors also found that adding a dedicated learnable sink token during pre-training further improves streaming. The limit is explicit in their own evaluation: retrieval accuracy drops to zero once the answer is further back than the cache. StreamingLLM makes a model run forever; it does not make it remember.

Heavy hitters: H2O and Scissorhands

Zhang et al. (2023) observed that attention in trained LLMs is over 95% sparse at inference and that a small set of tokens, which they call heavy hitters, receives most of the attention mass, correlated with frequent co-occurrence in the text. H2O scores each cached token by its accumulated attention and keeps the top scorers plus a recent window. With $A_{t,j}$ the attention weight from query position $t$ to cached token $j$ in a given head,

and when the cache exceeds $B_k$, the non-recent token with the lowest ${\sigma }_t$ is evicted, one per step, independently per head. The paper casts eviction as dynamic submodular maximisation and proves a guarantee for the greedy policy under mild assumptions. With 20% of the cache kept, it reports up to 29× higher throughput than offloading-based inference systems on OPT-6.7B and OPT-30B and up to 1.9× lower latency at the same batch size. Scissorhands (Liu et al., 2023) rests on a related "persistence of importance" hypothesis, that tokens important at one step stay important, and reports up to 5× cache reduction without quality loss, and 20× combined with 4-bit quantisation.

Prompt-aware and per-head policies

Two refinements matter for serving. SnapKV (Li et al., 2024) notes that each head's important prompt positions can be predicted from an observation window at the end of the prompt, so it compresses the prompt's cache once after prefill, keeping clustered positions per head. It reports 3.6× faster generation and 8.2× better memory efficiency at 16k-token inputs with comparable accuracy on 16 long-sequence datasets, and processing of up to 380k tokens on one A100-80GB. FastGen (Ge et al., 2023) profiles heads and applies a different policy to each: local windows for heads that attend locally, special-token-only caches for heads that attend to punctuation and special tokens, and full caches for heads that attend broadly.

Toy run: three eviction policies on a real model

Using the same Qwen2.5-0.5B-Instruct implementation, we decoded the first 2,048 tokens of the passage one token at a time with teacher forcing, so every prediction saw only what the cache policy had kept, and measured perplexity over all 2,047 predictions. The first check: token-by-token decoding with a full cache reproduced the full-sequence forward pass (perplexity 31.9153 against 31.9153 on a 101-token test). Each policy kept a fixed budget per key-value head per layer and evicted one entry per step once full: a recent window only; four sink tokens plus a recent window; and H2O-style accumulated-attention scores with half the budget reserved for recent tokens, the accumulation summed over the seven query heads sharing each key-value head. Keys were cached after the rotary embedding with their original positions, which is valid here because 2,048 tokens is well inside the model's trained context.

The full cache gave perplexity 16.10. A pure recent window collapsed at every budget: 100.0 with 256 entries (12.5% of the sequence), 120.7 with 128 and 174.3 with 64. Adding back just four initial tokens recovered almost all of it: 17.74, 19.15 and 20.58. H2O-style scoring was best at every budget: 16.98, 17.68 and 19.31, so keeping 12.5% of the cache cost 5.5% perplexity. This reproduces, on a different and post-trained model, both headline findings: the sink effect from StreamingLLM and the heavy-hitter advantage from H2O. The caveat from the failure-modes section applies in full. Running text is the easy case; we did not test retrieval of a specific fact from early in the passage, which evicting policies are designed to give up.

Interaction with post-trained models#

Calibrate on the fine-tune, not the base. Quantisation scales and outlier statistics are properties of activations, and fine-tuning moves them. A LoRA adapter on the key and value projections changes the cached values directly; a full fine-tune can change which channels carry outliers. FP8 scales or KVQuant datatypes calibrated on the base model can clip a fine-tune's keys. Recalibrate on samples of the fine-tuned model's real traffic, and when serving several adapters on one base, check the worst adapter, because the cache format is shared.

Errors compound with quantised weights. A model trained with QLoRA and served with 4-bit weights already carries weight-quantisation error; adding a quantised cache adds a second, correlated source. Evaluate the combination you will deploy, not each piece separately. See GPTQ and AWQ and FP8 and FP4 inference.

Long-context fine-tunes are the most sensitive. A model fine-tuned for document question answering, retrieval over long contracts, or multi-hop reasoning is valuable precisely because it uses distant tokens. Eviction removes them by design, and aggressive quantisation blurs the small logit differences that pick the right needle. These are the deployments where the memory savings are largest and the risk is highest.

Reasoning models and eviction. Models trained with GRPO or RLVR produce long chains of thought whose later steps refer back to earlier intermediate results. Eviction policies tuned on prompts may drop a number computed two thousand tokens earlier. Evaluate final-answer accuracy on your reasoning benchmarks with the eviction policy on.

Sinks can be trained in. StreamingLLM found that a dedicated sink token added during pre-training improves streaming. Fine-tuning cannot easily add that property to a model that lacks it, but a chat template that always starts with the same tokens gives a stable set of initial positions to keep as sinks.

Interaction with prefix caching and paging. Quantised caches work with paged blocks and prefix caching unchanged, since a block simply holds fewer bytes. Eviction does not: per-head eviction makes each head's cache a different subset of tokens, which breaks block-level sharing, and an evicted prefix cannot be reused by the next request. Most production engines therefore ship cache quantisation and leave eviction to research code or specialised long-context deployments.

Tuning knobs#

Failure modes and how to detect them#

Perplexity passes, retrieval fails. Perplexity on running text mostly measures local prediction, which recent tokens support. A compressed cache can keep perplexity within a few percent while losing the ability to quote a clause from 20,000 tokens back. Always include needle-style retrieval at your served context lengths and a task-level evaluation of the fine-tuned model.

NaNs or sudden garbage after a format change. FP8 casts of out-of-range values can produce NaN instead of saturating, as our toy run did. Symptoms are NaN logits or degenerate repetition on specific inputs. Check scale calibration coverage and clamping.

Outlier-driven collapse at low bits. Per-token integer quantisation of keys can collapse a model, as INT4 per-token did here (perplexity 302). If a low-bit format performs far worse than published numbers, check the grouping axis and whether keys are quantised before or after the rotary embedding.

Memory saved but no speedup. Fake-quantised or unpacked caches save nothing; the savings need kernels that read packed low-bit data and dequantise on the fly. Measure bytes per step and TPOT, not just reported cache size.

Losing the beginning of a conversation. Window-only eviction drops the system prompt first. Instruction following degrades over long sessions in ways that look like model drift. Keep sinks and the system prompt pinned.

When to use what#

Minimal implementation#

A KIVI-style quantiser and an H2O-style eviction step, as used in the toy runs, in PyTorch. Production engines fuse dequantisation into the attention kernel and keep integers packed.

import torch

def quantize(x, bits, dim):
    """asymmetric min-max over `dim`; returns integer codes, scale, zero"""
    mn, mx = x.amin(dim, keepdim=True), x.amax(dim, keepdim=True)
    scale = (mx - mn).clamp(min=1e-8) / (2 ** bits - 1)
    q = ((x - mn) / scale).round().clamp(0, 2 ** bits - 1).to(torch.uint8)
    return q, scale, mn

def dequantize(q, scale, zero):
    return q.float() * scale + zero

def kivi_compress(k, v, bits=2, group=32, residual=128):
    """k, v: (kv_heads, T, head_dim). Keys per-channel over token groups, values per-token.
    The last `residual` tokens (and any incomplete group) stay in full precision."""
    T = k.shape[1]
    n_q = max(0, (T - residual) // group * group)
    kq = k[:, :n_q].unflatten(1, (-1, group))           # (H, T/g, g, D): reduce over tokens in a group
    keys = quantize(kq, bits, dim=2)
    vals = quantize(v[:, :n_q], bits, dim=-1)           # reduce over head_dim: one scale per token
    return keys, vals, (k[:, n_q:], v[:, n_q:])

def h2o_evict(K, V, acc, budget, recent):
    """K, V: (kv_heads, n, D); acc: (kv_heads, n) accumulated attention. Drop one token per head."""
    n = K.shape[1]
    if n <= budget:
        return K, V, acc
    drop = acc[:, : n - recent].argmin(dim=1)             # lowest score outside the recent window
    keep = torch.ones_like(acc, dtype=torch.bool)
    keep[torch.arange(K.shape[0]), drop] = False
    H, D = K.shape[0], K.shape[2]
    return K[keep].view(H, n - 1, D), V[keep].view(H, n - 1, D), acc[keep].view(H, n - 1)

# each decode step, per layer: p = softmax(q K^T / sqrt(d)) of shape (kv_heads, group, n)
# acc += p.sum(1); then K, V, acc = h2o_evict(K, V, acc, budget, recent)
# StreamingLLM variant: always drop index `sinks` (the oldest non-sink token) instead of argmin

# production FP8 cache in vLLM:
# llm = LLM(model=path, kv_cache_dtype="fp8")   # scales default to 1.0 unless calibrated

In POFE, the long-context evaluation suite attached to a fine-tune can be rerun with the serving cache format, so a regression from an FP8 or INT4 cache shows up as a number next to the model rather than as a complaint after launch.

Further reading#

Liu et al. (2024), KIVI, explains why keys and values need different grouping and gives the residual-window design. Hooper et al. (2024), KVQuant, covers pre-RoPE keys, non-uniform datatypes and sparse outliers for 3-bit and lower caches. Xiao et al. (2023), StreamingLLM, introduces attention sinks and is careful about what streaming does not provide. Zhang et al. (2023), H2O, and Liu et al. (2023), Scissorhands, are the foundational eviction papers. Li et al. (2024), SnapKV, and Ge et al. (2023), FastGen, show prompt-aware and per-head policies. The vLLM quantised KV cache documentation describes the production FP8 path. For the architectural route to smaller caches, see the KV cache article on GQA and MLA.

Frequently asked questions#

References#

1. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu (2024). KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arxiv.org/abs/2402.02750
2. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. arxiv.org/abs/2401.18079
3. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis (2023). Efficient Streaming Language Models with Attention Sinks. arxiv.org/abs/2309.17453
4. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen (2023). H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. arxiv.org/abs/2306.14048
5. Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava (2023). Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time. arxiv.org/abs/2305.17118
6. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen (2024). SnapKV: LLM Knows What You are Looking for Before Generation. arxiv.org/abs/2404.14469
7. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao (2023). Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. arxiv.org/abs/2310.01801
8. vLLM project (2025). vLLM documentation: Quantized KV Cache. docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html
9. Qwen Team (2024). Qwen2.5-0.5B-Instruct model card. huggingface.co/Qwen/Qwen2.5-0.5B-Instruct

Related reading