Prefill/decode disaggregation, explained

Prefill/decode disaggregation runs the two phases of every LLM request on different GPUs. A prefill instance processes the prompt, produces the first token and the KV cache, then ships that cache over the network to a decode instance, which generates the remaining tokens. The two pools are sized, parallelised and sometimes even built from different hardware independently. It is the most structural of the scheduling techniques in this series: continuous batching keeps one GPU's batch full, chunked prefill limits how much the phases disturb each other on that GPU, and disaggregation stops them sharing a GPU at all. It is what the largest deployments use, and it is also easy to deploy in a way that is slower than the setup it replaced. This article covers both halves of that statement.

Where it came from#

Three groups published the idea within weeks of each other in late 2023 and early 2024. Splitwise (Patel, Choukse, Zhang, Shah, Goiri, Maleki and Bianchini, 2023) came from a characterisation of production traces: prompt computation is compute-intensive, token generation is memory-intensive and does not need the newest GPU, so the phases could run on different machine types. They reported 1.4 times higher throughput at 20% lower cost, or 2.35 times more throughput at the same cost and power. DistServe (Zhong, Liu, Chen, Hu, Zhu, Liu, Jin and Zhang, OSDI 2024) framed the problem as goodput, the highest request rate a system can serve while meeting its TTFT and TPOT SLOs for a target fraction of requests, and showed that colocation couples two things that should be chosen separately: the resources given to each phase and the parallelism used for each. DistServe reported serving 7.4 times more requests, or meeting 12.6 times tighter SLOs, than colocated systems of the time while keeping over 90% of requests within their latency targets. TetriInfer (Hu et al., 2024) combined disaggregation with fixed-size prefill chunks and load-aware decode placement, using 38% fewer resources while cutting average TTFT by 97%.

Production reports followed. Mooncake (Qin et al., 2024), the serving system behind Moonshot AI's Kimi, is built around a disaggregated, KV-cache-centric design that also pools KV in CPU DRAM and SSD; it reports up to 525% more throughput in simulated long-context scenarios and 75% more requests on real workloads. The DeepSeek-V3 technical report (DeepSeek-AI, 2024) describes prefill and decode deployed separately with different parallelism: the smallest prefill unit is 4 nodes with 32 GPUs, the smallest decode unit 40 nodes with 320 GPUs. Open-source engines followed: vLLM ships disaggregated prefill as an experimental feature with pluggable KV connectors (NIXL, LMCache, Mooncake), and SGLang has a PD disaggregation mode.

The bottleneck: two workloads, one configuration#

Take the roofline step-time model used throughout this series, with $N$ parameters in 16-bit precision, achievable bandwidth $B_W$ and achievable compute $F$. A prefill of $p$ tokens costs about $2Np/F$ seconds and is compute-bound for any $p$ above the ridge $F/B_W$ (about 221 tokens with our H100-class assumptions). A decode step for $b$ sequences with contexts $c_i$ and $\kappa$ KV bytes per token costs about $(2N+\kappa {\sum }_i{c}_i)/B_W$ and is bandwidth-bound. Colocation forces three compromises.

Interference. Any prefill in the same iteration as decodes delays them. Chunking bounds the delay but cannot remove it, and the bound you need for a tight TPOT target forces small chunks that slow prefill.

One parallelism plan. Prefill latency benefits from tensor parallelism across many GPUs because it is compute-bound; decode throughput benefits from replicas, large batches and expert parallelism. DistServe's central argument is that the best plan for one phase is rarely the best for the other. DeepSeek-V3's 32-GPU prefill and 320-GPU decode units are an extreme example.

One resource ratio. The fraction of GPU time each phase needs depends on the workload. With mean prompt $\bar{p}$ and mean output $\bar{o}$, the prefill share of GPU time at a full decode batch is

where $t(b)/b$ is the GPU time per decoded token and $n$ is the number of GPUs. Disaggregation has to pick $n_P:n_D$ explicitly. Colocation picks it implicitly, every iteration, which is an advantage disaggregation gives up.

The price: moving the KV cache

The decode instance needs the full KV cache of the prompt: $p\kappa$ bytes, where $\kappa =2\times n_{\text{layers}}\times n_{\text{kv heads}}\times d_{\text{head}}\times$ bytes per element. The transfer adds $T_{\text{xfer}}=T_0+p\kappa /B_{\text{link}}$ to the time before the second token. The useful comparison is with the prefill itself, because a layer-by-layer pipelined transfer can overlap with prefill compute:

The prompt length cancels. What matters is KV bytes per parameter, and grouped-query attention has made that small.

The ratios use the same 593 TFLOP/s achievable compute per GPU; with tensor parallelism over $g$ GPUs both the compute and the number of links scale by $g$, so the ratio holds. The 50 GB/s column is roughly one 400 Gb/s RDMA NIC per GPU; 450 GB/s is the order of intra-node NVLink. DistServe's analysis of OPT-66B, a full multi-head attention model, found 1.13 GB of KV for a 512-token request and 90 Gbps of transfer at 10 requests per second, which is why it placed prefill and decode stages on the same node when cross-node bandwidth was only 25 Gbps. Modern GQA and MLA models (DeepSeek's multi-head latent attention compresses KV further) make cross-node transfer comfortable. KV compression and FP8 KV halve it again.

How it works, step by step#

1. Route. A proxy receives the request and picks a prefill instance (shortest queue, or one holding a cached prefix) and a decode instance (most free KV memory).
2. Prefill. The prefill instance runs the prompt, usually in large batches since nobody is decoding there, writes KV blocks and samples the first token, which can be streamed to the client immediately.
3. Transfer. The KV blocks are sent to the decode instance, ideally layer by layer while prefill is still running, using RDMA or NVLink through a transfer library. The decode instance allocates blocks in its own paged cache to receive them.
4. Decode. The decode instance adds the sequence to its continuous batch, with no prefill ever in the way, and runs until a stop condition.
5. Release. Prefill KV blocks are freed once acknowledged (or kept as a prefix cache); decode blocks are freed at the end.

# Pseudocode of the proxy logic in a 1-hop disaggregated deployment
async def handle(req):
    p = pick(prefill_pool, key=lambda i: i.queued_tokens - i.prefix_hit(req.prompt))
    d = pick(decode_pool,  key=lambda i: -i.free_kv_blocks)
    first, kv_handle = await p.prefill(req.prompt, max_tokens=1, kv_target=d.id)
    yield first                                   # TTFT is set here
    async for tok in d.decode(req, kv_handle):    # waits for the KV transfer, then streams
        yield tok

Worked example: sizing the split#

Our workload: mean prompt 1,000 tokens, mean output 250, an 8B model, one GPU per instance. From the step-time model, the mean prefill takes 28.6 ms and each decoded token at a full batch of 256 costs 0.0807 ms of GPU time, so a request needs $250\times 0.0807\approx 20.2$ ms of decode time:

So 5 prefill and 3 decode GPUs, not the intuitive 4 and 4, and certainly not few prefill GPUs feeding many decoders. With longer prompts (mean 3,600 tokens actually sampled in our long-prompt workload), the prefill share rises to 0.67, or 5.3 of 8 GPUs. The transfer adds $1,000\times 131,072/50\times 10^9=2.6$ ms plus setup per request at 50 GB/s, about 10% of the prefill time.

Simulated numbers: when does it win?#

We extended the seeded serving simulator from the continuous batching and chunked prefill articles to 8 GPUs, each holding a full copy of an 8B model. Colocated configurations run 8 chunked-prefill replicas with round-robin routing. Disaggregated configurations run $n_P$ prefill-only instances (prompt batches up to 8,192 tokens) and $n_D$ decode-only instances (up to 256 sequences), with a KV transfer of 1 ms plus $p\kappa /50$ GB/s and decode placement on the least-loaded instance. Goodput is the highest tested arrival rate (in steps of 40 requests per second) at which at least 90% of requests meet both SLOs. Arrivals are Poisson, prompts and outputs lognormal. This is a model; it captures the phase ratio and interference, and it cannot capture the parallelism and hardware advantages that drive the largest published gains.

The results did not flatter disaggregation, and that is the point worth internalising.

• Standard SLO (TTFT 1 s, TPOT 30 ms). Colocated with a 512 budget reached 240 requests per second (optimistic kernels) or 160 (pessimistic, where decode attention does not overlap prefill compute). The best disaggregated split, 5P + 3D, reached 160 in both. The intuitive 4P + 4D reached 120, and 3P + 5D only 80, because prefill was the bottleneck: its p99 TTFT hit 3.9 s at 120 requests per second while decode GPUs idled at a median TPOT of 9 ms.
• Tight TPOT (TTFT 1 s, TPOT 10 ms). 3P + 5D and 4P + 4D both held 80 requests per second in both kernel models. Colocated 512 held 80 only with optimistic kernels; with pessimistic kernels it met the target for 82% of requests at that rate. A budget of 2,048 failed the target at every tested rate.
• Long prompts (mean 3,600 tokens). Colocated 512 reached 60 requests per second under the standard SLO; the best split, 5P + 3D, reached 40.

Three conclusions carry over to real deployments. First, for a small model that fits on one GPU, with modern chunked prefill, disaggregation is not a throughput feature, which is exactly what the vLLM documentation says: disaggregated prefill does not improve throughput; it lets you tune TTFT and inter-token latency separately and control tail ITL. Second, it shines when the TPOT target is tight relative to what chunking can deliver, because decode GPUs never see a prefill. Third, the split ratio is a first-order parameter that has to track the workload; a fixed split tuned for one prompt length is wrong for another. The large published wins come from regimes our toy omits: models that need multi-GPU parallelism where the best plans differ per phase (DistServe), very long contexts with KV pooling and reuse (Mooncake), and heterogeneous or power-limited hardware (Splitwise).

Interaction with fine-tuned and post-trained models#

Both pools must serve the same weights. A request prefilled on one checkpoint and decoded on another produces garbage silently: the KV cache is a function of the weights. When you roll out a new fine-tune, roll prefill and decode instances together and include the model version in the KV transfer handshake. The same holds for quantisation: a prefill instance with FP8 weights and a decode instance with BF16 weights produce a KV cache the decoder did not compute.

LoRA adapters. The adapter must be active on both sides, since the prompt's keys and values depend on it. In multi-LoRA deployments that doubles the adapter-placement problem; routers that are adapter-aware on both pools avoid paying load latency twice. Merged fine-tunes avoid the issue (merge vs adapter).

Output length shifts the ratio. Post-training changes output length dramatically. A reasoning model after GRPO may emit ten times the tokens of its SFT parent, moving the decode share from 41% to 80% or more; a short-answer classifier fine-tune moves it the other way. Re-derive $n_P:n_D$ whenever a new checkpoint changes the length distribution, and prefer deployments that can reassign instances between roles.

Custom templates and long system prompts. Fine-tunes that keep a long fixed system prompt are ideal for prefix caching on the prefill pool; routing requests with the same prefix to the same prefill instance turns most of the prompt into a cache hit and shifts the ratio toward decode.

Tuning knobs#

Failure modes and how to spot them#

Wrong ratio. One pool saturates while the other idles. Watch utilisation per pool and queue length at the prefill tier; a TTFT wall at modest load with low decode TPOT means too few prefill GPUs (Figure 3).

Transfer bottleneck. Rising gap between first and second token, with prefill and decode both healthy. Check NIC utilisation, whether transfers cross nodes unexpectedly, and whether layer-wise pipelining is enabled.

KV memory exhaustion on decode. Long outputs from reasoning fine-tunes fill decode KV and stall admissions; transferred caches wait with nowhere to land. Watch decode free-block counts and transfer queueing.

Version skew. Quality drops right after a rollout with no error: prefill and decode on different checkpoints. Pin versions in routing.

Short-request overhead. For very short prompts the fixed transfer setup dominates and an extra network hop costs more than the interference it avoids. Some systems keep a colocated path for short requests.

When to use it, and what to use instead#

The cost side of this decision, GPUs per million tokens at a given SLO, is treated in cost per million tokens and utilisation and batching economics.

Implementation sketch#

In vLLM, disaggregation is configured through a KV connector on each instance and a proxy in front. A minimal two-instance setup on one node:

# prefill instance (producer of KV)
vllm serve path/to/finetuned-model --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'
# decode instance (consumer of KV)
vllm serve path/to/finetuned-model --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'
# a proxy sends each request to 8100 with max_tokens=1, then to 8200 with the KV handle

SGLang's equivalent uses --disaggregation-mode prefill or decode with a transfer backend and a router. Treat the exact flags as version-specific; the feature is marked experimental in vLLM and the connector set changes between releases. Before deploying, run the same goodput sweep we ran here on your own traffic: colocated chunked prefill at two budgets against two or three splits, with your real SLOs. In POFE the serving recipe for a fine-tune starts colocated and only proposes a split when the measured goodput under your SLO says it pays.

Further reading#

Zhong et al. (2024) is the clearest argument for disaggregation and introduces goodput as the metric. Patel et al. (2023) make the hardware and power case. Hu et al. (2024) combine it with chunking and smarter decode placement. Qin et al. (2024) describe a production system built around the KV cache, and the DeepSeek-V3 report shows the extreme of per-phase parallelism. For the colocated alternative, read Agrawal et al. (2024) on Sarathi-Serve.

Frequently asked questions#

References#

1. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv.org/abs/2401.09670
2. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini (2023). Splitwise: Efficient generative LLM inference using phase splitting. arxiv.org/abs/2311.18677
3. Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, et al. (2024). Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. arxiv.org/abs/2401.11181
4. Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arxiv.org/abs/2407.00079
5. DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arxiv.org/abs/2412.19437
6. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arxiv.org/abs/2403.02310
7. vLLM project (2025). vLLM documentation: Disaggregated Prefilling (experimental). docs.vllm.ai/en/latest/features/disagg_prefill.html
8. SGLang project (2025). SGLang documentation: PD Disaggregation. docs.sglang.ai/advanced_features/pd_disaggregation.html

Related reading