The roofline of LLM inference: prefill, decode, TTFT and TPOT

The roofline of LLM inference is the one-page model that explains almost every serving decision: why the first token takes a different kind of time from every token after it, why batching is nearly free until it suddenly is not, why a 4-bit model can generate faster without computing faster, and why long contexts eat throughput. A transformer forward pass is a fixed number of floating-point operations and a fixed number of bytes moved from GPU memory. Whichever of those two takes longer on your hardware sets the time. Prefill, which processes the whole prompt at once, does hundreds or thousands of operations per byte and runs into the compute ceiling. Decode, which produces one token per sequence per step, does a handful of operations per byte and runs into the memory-bandwidth ceiling. This article derives both from first principles, puts real numbers on them for an 8B model on one H100, checks the shape against a measurement, and maps every major serving optimisation to the term of the equation it attacks. It is the entry point to the KV cache, batching and speculative decoding articles in this series, and the reason post-training choices such as adapter rank, quantisation and output length show up on the serving bill.

Two phases with different physics#

Autoregressive generation splits into two phases. In prefill the model reads the prompt of $n$ tokens in one forward pass, computes the attention keys and values for every position, stores them in the KV cache, and samples the first output token. In decode the model runs one forward pass per new token per sequence, each pass attending over everything cached so far and appending one new key and value per layer. A 2,000-token prompt with a 300-token answer is therefore one large matrix-matrix pass followed by 299 small matrix-vector passes over the same weights.

The two phases stress different parts of the chip. The observation goes back at least to Shazeer (2019), who attributed slow incremental decoding to the memory bandwidth spent repeatedly loading the large key and value tensors, and proposed multi-query attention to shrink them. Pope et al. (2022) built the first careful analytical model of transformer inference cost on accelerators, separating the latency of processing input tokens from the latency of generating output tokens, and reported 76% model FLOPs utilisation during large-batch prefill against 29 ms per generated token at small batch on PaLM 540B with int8 weights. The tool they and almost everyone since have used to reason about this is the roofline model of Williams, Waterman and Patterson (2009), originally written for multicore CPUs and now the standard lens for accelerator kernels (Yuan et al., 2024, survey LLM inference entirely through it).

The roofline model in one equation#

A kernel performs $F$ floating-point operations and moves $Q$ bytes between off-chip memory (HBM on a GPU) and the chip. Its arithmetic intensity is $I=F/Q$, in FLOP per byte. A device has a peak compute rate $\pi$ (FLOP/s) and a peak memory bandwidth $\beta$ (byte/s). If compute and data movement overlap perfectly, the kernel cannot finish faster than either resource allows:

The two bounds cross at the ridge point $I^{\star }=\pi /\beta$. Below it the kernel is memory-bound: doubling the arithmetic is free, halving the bytes halves the time. Above it the kernel is compute-bound: shrinking bytes buys nothing and only fewer FLOPs or faster arithmetic help. For an H100 SXM, the vendor datasheet lists 1,979 TFLOP/s of BF16 tensor throughput with sparsity, so 989 TFLOP/s dense, and 3.35 TB/s of HBM bandwidth. The ridge is

Any kernel that does fewer than about 295 BF16 operations per byte it reads is waiting on memory. The question for LLM serving is where prefill and decode land relative to 295.

Counting FLOPs and bytes in a forward pass#

Let the model have $P$ parameters stored at $b_w$ bytes each, $L$ layers, $n_h$ query heads, $n_{kv}$ key-value heads, head dimension $d_h$, and a KV cache stored at $b_{kv}$ bytes per element. Following Kaplan et al. (2020), a forward pass costs about $2P$ FLOPs per token for the matrix multiplies (one multiply and one add per weight), plus attention. The KV cache costs $\kappa$ bytes per token:

where the leading 2 counts keys and values. The full derivation and its dependence on GQA and MLA are in the KV cache article.

Decode

One decode step for a batch of $B$ sequences, each with $S$ tokens of context, reads every weight once (shared by the whole batch) and reads each sequence's KV cache once. Per layer and per sequence, attention costs $2n_h{d}_{h}S$ FLOPs for the scores $Q{K}^{\top }$ and another $2n_h{d}_{h}S$ for the weighted sum with $V$. Summing:

Two limits make this readable. With short contexts the weights dominate the bytes and $I_{\text{dec}}\approx 2B/b_w$, which is $B$ in BF16: one FLOP per byte per sequence in the batch. With long contexts the cache dominates and the intensity saturates at

which does not depend on batch size at all. Batching amortises weight reads because all sequences share the weights; it cannot amortise cache reads because every sequence has its own cache. For Llama 3 8B, with 32 query heads and 8 key-value heads in BF16, $I_{\text{attn}}=4$. A model with full multi-head attention sits at 1. Storing the cache in FP8 doubles it. These numbers are two orders of magnitude below the H100 ridge, which is why the decode attention kernel is memory-bound on every current GPU no matter how it is written.

Prefill

Prefill processes $n$ prompt tokens together. With causal masking each head's score and value products cost about $n^2{d}_h$ multiply-adds each, so

A 512-token prompt already has intensity above 500, past the ridge. Prefill is compute-bound for any realistic prompt, and its time scales with the number of prompt tokens rather than with the weight size.

TTFT, TPOT and what users feel#

The two phases map onto the two latency numbers every serving SLO uses. Time to first token (TTFT) is queueing plus prefill. Time per output token (TPOT, also called inter-token latency when measured per gap) is one decode step. End-to-end latency for $n_{\text{out}}$ output tokens is

With the roofline we can bound both. Write $\text{MFU}$ for model FLOPs utilisation (achieved FLOP/s over peak) and $\text{MBU}$ for model bandwidth utilisation (achieved bytes/s over peak); for planning, 50% MFU in prefill and 70% MBU in decode are reasonable starting assumptions to replace with your own measurements (Pope et al. report 76% MFU for large-batch prefill on TPUs). Then

The TPOT expression is the single most useful formula in LLM serving. It says that generating one token for one user costs a full read of the weights, that adding users is nearly free while weights dominate the bytes, and that every token of context any user holds adds $\kappa$ bytes to every step for everyone in the batch.

Worked example: Llama 3 8B on one H100#

The Llama 3 paper lists the 8B model at 32 layers, model dimension 4,096, 32 attention heads and 8 key-value heads, so $d_h=128$. Take $P=8.03\times 10^9$, BF16 weights and cache ($b_w=b_{kv}=2$).

Bytes. Weights: $2\times 8.03\times 10^9=16.06$ GB. KV cache per token: $\kappa =2\cdot 32\cdot 8\cdot 128\cdot 2=131,072$ bytes, exactly 128 KiB. A 2,048-token context is 268 MB per sequence.

Batch-1 decode. $Q\approx 16.06$ GB plus 0.27 GB of cache, $F\approx 2\times 8.03\times 10^9=16$ GFLOP. Time bound: $16.33\text{ GB}/3.35\text{ TB/s}=4.87$ ms if bandwidth were fully used, against 0.02 ms for the arithmetic. The GPU's tensor cores are idle for 99.5% of the step. The ceiling is about 205 tokens per second for a single user, and nothing short of moving fewer bytes changes it.

Prefill of 2,000 tokens. $F\approx 2\cdot 8.03\times 10^9\cdot 2000+2\cdot 32\cdot 32\cdot 128\cdot 2000^2=32.1+1.0=33.1$ TFLOP (attention is only 3% at this length). At 100% of 989 TFLOP/s that is 33.5 ms; at a realistic 50% MFU, about 67 ms. Bytes barely matter: 16 GB at 3.35 TB/s is 4.8 ms, hidden under the compute.

Batch 32, about 2,150 tokens of context each. Weights 16.06 GB plus $32\times 2150\times 131,072=9.0$ GB of cache: 25.1 GB per step, 7.49 ms at full bandwidth, and 32 tokens per step gives about 4,270 tokens per second. Batching 32 users cost 54% more time per step and delivered 21 times the throughput.

End to end. One user in that batch with a 2,000-token prompt and 300 output tokens sees $67+299\times 7.49\approx 2,307$ ms, ignoring queueing. Decode is 97% of the wall time, which is why long-output workloads such as reasoning models are dominated by memory bandwidth even when their prompts are short.

Capacity. An 80 GB H100 minus 16 GB of weights minus a few GB of activations and runtime leaves about 60 GB for cache. The maximum batch is

which gives 223 sequences at 2,048 tokens, 56 at 8,192, 14 at 32,768 and 3 at 131,072. The decode intensity at $B=223,S=2048$ is about 50 FLOP/byte, still a sixth of the ridge. On this hardware, an 8B BF16 model never reaches the compute roof in decode: memory capacity caps the batch long before arithmetic intensity gets there.

Checking the shape: a measured knee#

The roofline is a bound, so it is worth seeing whether real hardware follows it. We timed a single FP16 matrix multiply with the shape of Llama 3 8B's MLP up-projection (4,096 by 14,336, 117 MB of weights) against batches of 1 to 4,096 input rows, on the GPU of an Apple M4 Max laptop through PyTorch's MPS backend, taking the best of seven runs of twenty calls after warm-up (seed 0). This is not an H100, but the roofline does not care which chip it describes, only where its ridge sits.

At batch 1 one call took 0.274 ms, an effective 429 GB/s of weight streaming and 0.43 TFLOP/s of arithmetic. At batch 16 it took 0.425 ms: sixteen times the work for 1.55 times the time, with arithmetic up to 4.4 TFLOP/s. The best rate was 7.59 TFLOP/s at batch 64 (0.991 ms). Measured bandwidth and peak put this chip's ridge near $7.59\times 10^{12}/4.29\times 10^{11}\approx 18$ FLOP/byte, and since a BF16 or FP16 matmul with $B$ rows has intensity just under $B$, the model predicts the knee between batch 16 and 64. That is where the measured curve bends. Above batch 128 this backend's kernel got slower per FLOP (3.4 TFLOP/s at 256), a kernel-selection effect rather than a roofline one, so the chart stops at 128. The lesson transfers directly: on an H100 the same knee sits near batch 300 for weight-bound layers, which is why decode servers batch aggressively.

What post-training changes on the roofline#

Every serving cost in the formulas above is shaped by choices made during post-training, which is why it pays to look at them before a fine-tune ships rather than after.

Merged fine-tunes have exactly the base model's roofline. A full fine-tune or a merged LoRA changes the values in the weight matrices, not their shapes, so $P$, $b_w$ and $\kappa$ are unchanged.

Unmerged adapters add bytes and kernels. A rank-16 LoRA on all linear layers of an 8B model is about 42M parameters, 84 MB in BF16, 0.5% of the weight bytes read per step, so its bandwidth cost is small. The real costs are the extra small matmuls per layer, which add launch overhead that matters most in the latency-bound small-batch regime, and, when a batch mixes many adapters, one extra read per distinct adapter per step. The multi-LoRA serving and merge versus adapter articles work through these.

Quantised fine-tunes change $b_w$. A model trained with QLoRA and served with 4-bit weights cuts the batch-1 decode bound from 4.8 ms to 1.2 ms for 8B, a 4× ceiling on single-stream speed. Prefill does not get faster from weight-only quantisation because it is compute-bound and the weights are dequantised to BF16 for the matmul; only lower-precision arithmetic, such as FP8 tensor cores at 1,979 dense TFLOP/s on H100, raises $\pi$. See GPTQ and AWQ and FP8 and FP4 inference.

Chat templates and system prompts add prompt tokens to every request, and TTFT is linear in $n$ (quadratic for very long prompts). A 1,500-token system prompt baked into a fine-tuned product costs about 50 ms of H100 prefill per request on an 8B model at 50% MFU, unless prefix caching reuses it.

Output length is the largest lever post-training has on serving cost. A model trained with GRPO or RLVR to reason at length may emit thousands of tokens per answer. Each one pays a decode step, and each one grows $S$, which raises the cache bytes for every later step and lowers $B_{\max }$. A length penalty during RL is a serving optimisation.

Tokenizers and vocabularies change both $n$ and $P$. A larger vocabulary compresses text into fewer tokens but enlarges the embedding and output head: Llama 3's 128,000-entry vocabulary at width 4,096 is about 0.52B parameters in the output projection alone, roughly 6.5% of the bytes read per decode step for the 8B model.

Every optimisation is a term in the equation#

The value of the model is that it turns a zoo of serving techniques into a short list of terms. If a technique does not reduce the dominant term for your workload, it will not help.

Tensor parallelism deserves a number. Llama 3 70B (80 layers, 8 key-value heads) needs 141 GB of BF16 weights, so it does not fit on one 80 GB GPU. Split four ways, each GPU streams a quarter of the weights per step and the batch-1 bound drops from a hypothetical 42 ms to 10.5 ms, before the all-reduce after each attention and MLP block, which adds a latency floor that grows with the number of GPUs. Its cache is 320 KiB per token, 2.5 times the 8B model's, so capacity per GPU per sequence is worse even after the split.

Where the simple model misleads#

Forgetting the cache

Back-of-envelope estimates that use only $b_{w}P$ for decode bytes look fine in short-prompt benchmarks and fall apart in production, where retrieval contexts, agent histories and reasoning traces make $BS\kappa$ the larger term. In the worked example, at batch 64 with 8,192-token contexts the cache is 69 GB per step against 16 GB of weights. Symptom: TPOT rises with traffic even though the batch size is unchanged. Check by logging the total cached tokens per step alongside TPOT; the relationship should be close to linear.

Fixed overheads at small scale

For small models or small batches, a decode step is a few milliseconds, comparable to the cost of launching hundreds of kernels, running the scheduler in Python, sampling, and detokenising. These do not appear in the roofline. Symptom: measured TPOT for a 1B model is far above its $b_{w}P/\beta$ bound and does not improve with weight quantisation. Fixes are engineering, not algorithmic: CUDA graphs, fused sampling kernels, asynchronous scheduling.

Mixture-of-experts accounting

For an MoE model, FLOPs follow the active parameters but bytes do not. DeepSeek-V2 has 236B total and 21B active parameters per token. At batch 1 a decode step reads only the routed experts, about 42 GB in BF16; at large batches different tokens route to different experts and the step touches nearly all 472 GB. MoE models therefore look cheap at low batch on FLOPs and expensive on capacity, and their decode intensity per expert is lower than a dense model's at the same batch because each expert sees only a fraction of the tokens.

Datasheet peaks

Accelerator datasheets often quote throughput with 2:4 structured sparsity, which dense LLM inference does not use. Halve those numbers. Neither peak is attainable in practice either: a ridge computed from datasheet values is a guide to which side of the line you are on, not a prediction of exact time.

Prefill stalls decode

When one engine runs both phases, a long prompt admitted into a running batch makes that iteration compute-bound for tens or hundreds of milliseconds, and every decoding user sees a TPOT spike. Symptom: p99 inter-token latency far above p50, correlated with long-prompt arrivals. Chunked prefill bounds the stall; disaggregation removes it by running the phases on different GPUs, as DistServe (Zhong et al., 2024) proposed.

Throughput without a latency constraint

The throughput curve rises with batch size, and so does TPOT. Reporting peak tokens per second without the TPOT and TTFT at which it was achieved hides the trade. The useful number is goodput: requests per second served within the SLO. The latency SLO tuning article turns this into a procedure, and the economics series converts it into cost per million tokens.

Diagnosing your own deployment#

Use the model to classify the bottleneck before changing anything. Compute $b_{w}P$, $\kappa$ and $I^{\star }$ for your model and GPU, then compare measured numbers against the bounds.

The knobs that move you along the curve are the engine's limits on concurrent sequences and tokens per iteration, the fraction of GPU memory reserved for the cache, the weight and cache data types, and the tensor-parallel degree. Raise concurrency until TPOT hits the SLO; raise the cache fraction until the engine stops preempting requests; then decide whether the remaining bound is bytes or FLOPs and pick the lever from the table above.

A roofline calculator#

The whole model fits in a function. It reproduces the numbers in this article and is a better first estimate than most benchmarks, because it tells you which term to measure.

from dataclasses import dataclass

@dataclass
class Model:
    params: float      # total parameters read per step (active for small-batch MoE)
    layers: int
    q_heads: int
    kv_heads: int
    head_dim: int
    w_bytes: float = 2.0   # 2 = BF16, 1 = FP8/INT8, 0.5 = INT4
    kv_bytes: float = 2.0

@dataclass
class GPU:
    flops: float       # dense FLOP/s at the compute dtype
    bw: float          # HBM bytes/s
    mem: float         # bytes

def kv_per_token(m):
    return 2 * m.layers * m.kv_heads * m.head_dim * m.kv_bytes

def prefill_ms(m, g, n, mfu=0.5):
    flops = 2 * m.params * n + 2 * m.layers * m.q_heads * m.head_dim * n * n
    return 1e3 * max(flops / (g.flops * mfu), m.w_bytes * m.params / g.bw)

def decode_step_ms(m, g, batch, ctx, mbu=0.7):
    flops = 2 * m.params * batch + 4 * m.layers * m.q_heads * m.head_dim * ctx * batch
    nbytes = m.w_bytes * m.params + batch * ctx * kv_per_token(m)
    return 1e3 * max(flops / g.flops, nbytes / (g.bw * mbu))

def max_batch(m, g, ctx, other=4e9):
    return int((g.mem - m.w_bytes * m.params - other) // (ctx * kv_per_token(m)))

llama3_8b = Model(8.03e9, 32, 32, 8, 128)
h100 = GPU(flops=989e12, bw=3.35e12, mem=80e9)
print(prefill_ms(llama3_8b, h100, 2000))              # ~67 ms
print(decode_step_ms(llama3_8b, h100, 32, 2150, 1.0)) # ~7.5 ms at full bandwidth
print(max_batch(llama3_8b, h100, 8192))               # ~55 sequences

To measure the real numbers, the serving benchmarks bundled with open-source engines such as vLLM and SGLang replay a request trace and report TTFT, TPOT and inter-token latency percentiles. Divide measured TPOT into the bytes-per-step from decode_step_ms to get MBU; divide prefill FLOPs by measured prefill time to get MFU. If MBU is above 60% you are near the wall and only fewer bytes will help. In POFE, the same calculation sits behind the serving estimate attached to a fine-tuning run, so the cost of a longer template or a longer reasoning trace is visible before deployment.

Further reading#

Williams, Waterman and Patterson (2009) introduced the roofline model and remain the clearest statement of it. Pope et al. (2022) is the foundational analysis of transformer inference cost, including the prefill and decode split, partitioning across chips and the case for multi-query attention at long context. Shazeer (2019) identified KV bandwidth as the decode bottleneck. Kaplan et al. (2020) gives the $2N$ FLOPs-per-token accounting. Yuan et al. (2024) surveys inference optimisations through a roofline lens and ships an analysis tool. Zhong et al. (2024) shows why the two phases are better served separately. The Llama 3 report supplies the model shapes used here, and the H100 datasheet supplies the hardware numbers. Continue with the KV cache article for the memory side of the equation.

Frequently asked questions#

References#

1. Samuel Williams, Andrew Waterman, David Patterson (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. doi.org/10.1145/1498765.1498785
2. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean (2022). Efficiently Scaling Transformer Inference. arxiv.org/abs/2211.05102
3. Noam Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need. arxiv.org/abs/1911.02150
4. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei (2020). Scaling Laws for Neural Language Models. arxiv.org/abs/2001.08361
5. Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights. arxiv.org/abs/2402.16363
6. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv.org/abs/2401.09670
7. Llama Team, AI @ Meta (Aaron Grattafiori, Abhimanyu Dubey, et al.) (2024). The Llama 3 Herd of Models. arxiv.org/abs/2407.21783
8. DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arxiv.org/abs/2405.04434
9. NVIDIA (2024). NVIDIA H100 Tensor Core GPU (product specifications). www.nvidia.com/en-us/data-center/h100/

Related reading