Chunked prefill (stall-free batching), explained

Chunked prefill splits the processing of a long prompt into fixed-size pieces and runs each piece in the same forward pass as the single-token decode steps of the requests already generating. Instead of a 4,096-token prompt occupying the GPU for one 120 ms iteration while every other user waits for their next token, it is spread over eight iterations of about 16 ms, each of which also emits a token for everyone else. It is a refinement of continuous batching: the scheduler still rebuilds the batch every iteration, but now it also decides how many prompt tokens of each request to include, under a per-iteration token budget. vLLM enables it by default in its V1 engine, SGLang and TensorRT-LLM ship equivalent modes, and the single setting that controls it, the token budget, is one of the most consequential numbers in a serving configuration.

Where it came from#

The technique was introduced as chunked-prefills with decode-maximal batching in SARATHI (Agrawal, Panwar, Mohan, Kwatra, Gulavani and Ramjee, 2023). Their observation was that the two phases of a request use the GPU in opposite ways. A prefill iteration multiplies thousands of prompt tokens against the weights and saturates compute. A decode iteration multiplies one token per sequence against the same weights and is limited by memory bandwidth. If one prefill chunk is paired with as many decodes as fit, the decodes "piggyback" on weight reads the prefill was paying for anyway. SARATHI reported up to 10 times higher decode throughput and up to 1.33 times higher end-to-end throughput for LLaMA-13B on an A6000, and 6.29 times fewer pipeline bubbles with pipeline parallelism on GPT-3.

The follow-up, Sarathi-Serve (Agrawal et al., OSDI 2024), reframed the idea around latency. Iteration-level schedulers that prioritise prefills, like vLLM's original scheduler, create generation stalls: whenever a prefill is scheduled between two decode iterations, every running request's next token is delayed by the whole prefill, and the authors observed stalls lasting several seconds in vLLM. Schedulers that prioritise decodes, like FasterTransformer's request-level batching, avoid stalls but leave the batch small and throughput low. Stall-free scheduling with a token budget gets both. Sarathi-Serve reported 2.6 times higher serving capacity than vLLM for Mistral-7B on one A100 and up to 3.7 times for Yi-34B on two A100s under tail-latency constraints. In parallel, DeepSpeed-FastGen (Holmes et al., 2024) shipped the same idea under the name Dynamic SplitFuse, reporting up to 2.3 times higher effective throughput and 3.7 times lower tail latency than vLLM at the time.

The bottleneck: prefill-decode interference#

Use the roofline step-time model from the roofline article. For an iteration that processes $P$ prompt tokens and $D$ decode tokens, with $N$ parameters in 16-bit precision, achievable bandwidth $B_W$, achievable compute $F$, and KV traffic $K$ bytes:

The two phases sit on opposite sides of the ridge $P^{\ast }=F/B_W$. With 60% of 989 TFLOP/s and 80% of 3.35 TB/s (our assumption for an H100-class GPU), $P^{\ast }\approx 221$ tokens. A batch of 64 decodes at 1,125-token contexts takes 10.0 ms in this model, almost all of it memory traffic. A 4,096-token prefill takes 118 ms, almost all of it compute. Put them in the same iteration and the decodes cost almost nothing extra in the linear layers (they are 64 more rows in a 4,160-row matrix multiply), but they now have to wait 118 ms for their token.

The quantity users feel is the time between tokens (TBT), and its tail is what an SLO constrains. Without chunking, the worst-case gap for a running request is roughly one decode step plus the longest prompt admitted in between:

where $C$ is the token budget per iteration and $b$ the number of decoding sequences. The first grows with the longest prompt anyone sends; the second is bounded by a number you choose.

How it works, step by step#

Every iteration the scheduler fills a token budget $C$ in a fixed priority order:

1. Add one token for every running decode. Each costs one unit of budget.
2. If a prefill was already partly processed, add its next chunk: $\min (C_{\text{left}},p-p_{\text{done}})$ tokens starting at offset $p_{\text{done}}$.
3. Admit waiting requests in arrival order while budget, sequence slots and KV blocks remain, giving each as many prompt tokens as fit.
4. Run one forward pass over the flattened tokens. Prompt chunks write their keys and values into the KV cache and attend causally to everything cached for their request so far; decodes append one token.
5. A request whose last prompt chunk completes samples its first token in that same pass and becomes a decode in the next iteration.

def schedule(running, partial, waiting, budget, max_seqs, kv):
    batch, left = [], budget
    for s in running:                                  # 1. decodes first: never stall them
        batch.append((s, 1)); left -= 1
    for s in partial + waiting:                        # 2-3. continue partial prefills, then admit
        if left <= 0: break
        if s in waiting:
            if len(running) + len(partial) >= max_seqs or not kv.can_allocate(s): break
            waiting.remove(s); partial.append(s)
        n = min(left, s.prompt_len - s.prefilled)
        batch.append((s, n)); left -= n
    return batch                                       # 4. one ragged forward pass

The only new state is prefilled per request. The attention kernel must handle a query block of $n$ tokens attending to $p_{\text{done}}+n$ keys, partly read from the paged KV cache, which variable-length attention kernels such as FlashAttention's varlen and paged variants already do.

What chunking costs

Chunking is not free, and the costs set a floor on the budget.

Repeated KV reads. Chunk $k$ of a prompt (counting from 0) must read the $kc$ cached tokens before it. Over $n=p/c$ chunks the extra traffic is

with $\kappa$ the KV bytes per token. For a 4,096-token prompt in 512-token chunks on a model with $\kappa =131,072$ bytes (Llama-3.1-8B shape), that is $131,072\times 512\times 28=1.88$ GB, or 0.70 ms at 2.68 TB/s. Small, because attention over a prefill chunk is compute-bound anyway, as Sarathi-Serve also notes.

Losing the ridge. The real floor is the weight read. Each chunk re-reads all $2N$ bytes of weights, so a chunk smaller than $P^{\ast }$ is memory-bound and pays the full weight-read time for fewer than $P^{\ast }$ tokens of work:

In our model an 8,192-token prompt takes 251 ms in one shot. In 128-token chunks it takes 427 ms (70% overhead); at 256 the overhead is 6.2%, at 512 it is 3.0%, at 2,048 it is 0.6% (Figure 2). Running decodes fill some of the gap in a mixed batch, but the lesson stands: keep the budget comfortably above the ridge of your GPU and model.

Tile quantisation. Matrix multiplication kernels work in tiles, so a chunk of 257 tokens can cost much more than 256. Sarathi-Serve measured a 32% longer prefill for chunk size 257 than for 256. Use budgets that are multiples of 128 or 256 and remember that the decode tokens count against the budget, so the prefill share is $C-b$, not $C$.

Worked example#

Sixty-four users are decoding with 1,125-token contexts. A 4,096-token prompt arrives at $t=20$ ms. Numbers come from the step-time model with an 8B model on one H100-class GPU.

Without chunking (prefill-first). The next iteration is the whole prefill: 118 ms in which the 64 users receive nothing. Their gap is $118+10=128$ ms instead of the usual 10 ms. The new request's first token appears at $20+118=138$ ms.

With 512-token chunks. Each iteration carries the 64 decode tokens plus one 512-token chunk of the prompt (a budget of 576 once decodes are counted). Each mixed iteration takes 16.2 to 17.8 ms, rising slightly as the chunk attends to more cached prefix, and eight of them finish the prompt in 136 ms. The new user's first token arrives 18 ms later than without chunking (15% longer TTFT for this request), and in exchange the 64 existing users never wait more than 17.8 ms, a 7.2 times smaller worst-case gap. Their median time per output token also falls, because they keep producing tokens during the prefill instead of idling through it.

The arithmetic of the trade is general. Chunking moves latency from the many running requests to the one arriving request, and the exchange rate is favourable whenever more than a handful of requests are decoding.

Simulated numbers: choosing the budget#

We extended the seeded simulator from the continuous batching article with a chunked scheduler (decodes first, then partial prefills, then new admissions) and replayed Poisson arrivals with lognormal prompts (mean 1,000 tokens, up to 8,192) and outputs (mean 250). Because real kernels may or may not overlap decode attention with prefill compute, we ran two step-time models: an optimistic one in which memory traffic and compute in an iteration fully overlap, and a pessimistic one in which decode attention runs after the matrix multiplies and adds its full memory time. Everything below is model output, not a benchmark.

The p99 gap between tokens is set almost entirely by the budget: roughly 16, 30, 59 and 99 ms for budgets of 512, 1,024, 2,048 and 8,192, against 207 ms without chunking. Very small budgets buy smooth decoding with slower first tokens: at 256, p99 TTFT rises to 465 ms optimistic and 836 ms pessimistic, because each prompt needs many more iterations. That matches the vLLM documentation's guidance that smaller budgets (around 2,048) favour inter-token latency and larger ones (above 8,192) favour TTFT.

Throughput is where the two step-time models disagree, which is itself the useful finding. In the optimistic model, saturated throughput rose from 4,431 output tokens per second without chunking to 5,924 with a 512 budget (34% more), then fell to 5,182 at 2,048 and 4,874 at 256, the last because chunks near the ridge pay the weight read too often. In the pessimistic model every policy saturated between 4,373 and 4,496 tokens per second. Chunked prefill is reliably a latency tool; how much throughput it adds depends on how well your kernels overlap mixed batches, which is the gap POD-Attention (Kamath et al., 2024) targets with a fused prefill-decode attention kernel that speeds attention up by up to 59%.

Goodput, the highest arrival rate at which p99 TBT stays under 50 ms and p99 TTFT under 1 s, makes the choice concrete. Prefill-first failed the TBT target at every rate we tested (8 requests per second and up). Budgets of 512 and 1,024 sustained 24 and 26 requests per second in the optimistic model and 16 and 18 in the pessimistic one. A budget of 2,048 missed the 50 ms target by a few milliseconds at every rate but sustained 24 requests per second (optimistic) under a 100 ms target.

Interaction with fine-tuned and post-trained models#

Long fixed prompts. Fine-tuned agents and RAG models often carry long system prompts, tool schemas and retrieved context, so prefill dominates GPU time. Chunking keeps those prompts from stalling everyone else, and prefix caching removes the shared part entirely: cached blocks are skipped and only the uncached suffix is chunked. A common fine-tuning win is to train the behaviour into the weights and shorten the system prompt, which cuts prefill cost far more than any scheduler setting.

Reasoning models. Post-trained reasoning models generate thousands of tokens per request, so at any moment many sequences are decoding. That raises the stakes of stalls (more users per stall) and also means the decode tokens eat more of the budget. With 256 running sequences, a 512 budget leaves only 256 tokens for prefill, right at the ridge. Raise the budget with the running batch or cap max_num_seqs.

LoRA adapters. Chunked iterations mix tokens from different adapters in the same pass, which multi-LoRA kernels handle per row. Adapter compute scales with tokens, so a long prompt for a high-rank adapter makes its chunks slightly more expensive; budget-based scheduling absorbs this automatically. See multi-LoRA serving.

Prefill-only models. Reward models, classifiers and embedding models fine-tuned from an LLM never decode. Chunking has no one to protect there and only adds overhead; use large budgets or disable it. The same holds for offline scoring jobs such as computing log-probabilities for DPO data or evaluations.

Speculative decoding. Draft tokens being verified count against the same budget. With speculative decoding and $k$ speculative tokens, each running sequence uses $k+1$ budget units, which can starve prefills unless the budget grows accordingly.

Tuning knobs#

Sarathi-Serve's procedure is the right one: profile iteration time against token count for your model, GPU and parallelism once, then set the budget to the largest value whose iteration time stays within your TBT target. Then verify with a load test that replays your real length distribution and read the p99, not the mean. Our latency SLO tuning article covers the load-test side.

Failure modes and how to spot them#

Budget below the ridge. Throughput falls and TTFT rises while TBT looks excellent. Check that the prefill share $C-b$ is several hundred tokens at your typical running batch.

Head-of-line blocking among prefills. If a long prompt may take the whole prefill share, a 100,000-token document blocks every short prompt behind it for many iterations. Symptom: bimodal TTFT, with short prompts slow only when a long one is in flight. Cap the tokens a long prompt may take per iteration so short prompts can be admitted alongside it.

TTFT regressions after enabling it. Expected at small budgets. If TTFT matters more than smoothness, raise the budget or consider prefill/decode disaggregation, which removes the trade-off by giving prefills their own GPUs.

Mismatched expectations under pipeline parallelism. Chunking makes iterations uniform, which is what reduces pipeline bubbles. If you see bubbles anyway, check that prefill chunks are not being split across micro-batches unevenly.

When it helps, when it hurts#

Implementation sketch#

In vLLM V1 chunked prefill is on by default and the budget is the main control:

from vllm import LLM

llm = LLM(
    model="path/to/your-finetuned-model",
    enable_chunked_prefill=True,       # default in V1
    max_num_batched_tokens=1024,       # token budget per iteration: the TBT dial
    max_num_seqs=128,                  # decodes consume budget; keep C - b above the ridge
    enable_prefix_caching=True,        # skip shared system prompts entirely
)

SGLang exposes the same control as --chunked-prefill-size. To find your own ridge, time a forward pass for token counts from 64 to 8,192 with no decodes and plot time against tokens: the knee where time starts growing linearly is $P^{\ast }$. In POFE, served fine-tunes default to a budget picked from that profile and the target TBT, and the load test that checks it uses the prompt-length distribution from your own traffic.

Further reading#

Agrawal et al. (2023) introduced chunked prefill as a throughput technique, and Agrawal et al. (2024) is the essential paper on its latency role, with the token-budget methodology and the generation-stall analysis. Holmes et al. (2024) describe the equivalent Dynamic SplitFuse. Kamath et al. (2024) show why the attention kernel is the remaining inefficiency in mixed batches. Zhong et al. (2024) argue that past a certain SLO strictness, separating the phases beats mixing them, which is the subject of our disaggregation article.

Frequently asked questions#

References#

1. Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee (2023). SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arxiv.org/abs/2308.16369
2. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arxiv.org/abs/2403.02310
3. Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, et al. (2024). DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arxiv.org/abs/2401.08671
4. Aditya K Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, Ashish Panwar (2024). POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference. arxiv.org/abs/2410.18038
5. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv.org/abs/2401.09670
6. vLLM project (2025). vLLM documentation: Optimization and Tuning. docs.vllm.ai/en/latest/configuration/optimization.html

Related reading