Tuning for latency SLOs: batch size, token budgets, TTFT, TPOT and goodput

Tuning for latency SLOs means choosing the serving engine's scheduler settings (how many sequences run at once, how many tokens each step may process, how long outputs may grow, how prefill and decode share the GPU) so that the largest possible request rate is served while each request still meets its latency targets. The targets are usually two: time to first token (TTFT), which a user feels as responsiveness, and time per output token (TPOT), which sets streaming speed. The quantity to maximise is goodput: the highest arrival rate at which a target fraction of requests, say 90%, meets both. Throughput alone is the wrong objective, because a server can raise tokens per second by letting latency degrade without bound. In a post-training pipeline this is the last tuning loop, and it is not independent of the earlier ones: a fine-tune that answers twice as long, or a template that adds a long system prompt, changes the right settings.

The mechanics: one step, two kinds of work#

A continuous-batching engine (see continuous batching) runs in steps. Each step processes one new token for every running sequence in decode, plus some number of prompt tokens for sequences still in prefill. With $n_d$ decoding sequences, $n_p$ prefill tokens scheduled in the step, $P$ parameters, per-token KV bytes $\kappa$ and total context $C$ across the decoding sequences, a roofline estimate of step time (see the roofline) is

where $c_0$ is fixed per-step overhead (scheduling, sampling, launches), $\beta$ memory bandwidth and ${\pi }_{\text{eff}}$ achievable matmul throughput. A pure decode step is memory-bound and its time grows slowly with $n_d$, through the KV term. A step carrying a large prefill chunk is compute-bound and its time grows linearly with $n_p$. Every decoding sequence in that step waits for the whole step, so TPOT is set by the slowest steps a request lives through, and TTFT by how long a request queues plus how many steps its prompt needs. Two laws govern the rest. Little's law (Little, 1961) ties concurrency to rate and time in system,

so the engine must allow at least $\lambda {\bar{L}}_{\text{out}}\text{TPOT}$ concurrent sequences or requests queue. And stability: total work per second must stay below capacity, or the queue, and TTFT, grow without bound. Goodput, as defined by Zhong et al. in DistServe (2024), is

for SLOs ${\tau }_1,{\tau }_2$ and attainment target $q$.

Choosing the SLO before tuning for it#

The settings that maximise goodput depend on the targets, so the targets come first, and they should come from the product, not from a benchmark. Three questions settle most of it. Who consumes the tokens? A person reading a streamed answer needs a short TTFT and a TPOT comfortably faster than reading speed; 25 ms per token is 40 tokens per second, already faster than anyone reads, so a chat route can often accept 40 to 60 ms and spend the slack on throughput. An agent or tool chain that waits for the complete output cares about end-to-end time, $\text{TTFT}+L_{\text{out}}\cdot \text{TPOT}$, so long outputs make TPOT the dominant term and the TTFT target can be relaxed. A batch or evaluation route has no latency SLO at all and should run on separate capacity or at low priority. Which percentile? A 90% attainment target tolerates a slow tail that a 99% target does not; moving from p90 to p99 typically costs a large fraction of goodput, because the tail is made of the longest prompts and outputs. How is TPOT measured? Per-request mean inter-token time (used here and in DistServe) hides individual stalls; the p99 of individual inter-token gaps exposes them and penalises large prefill chunks much more. Pick one definition and use it in both load tests and production dashboards.

Different routes on the same deployment usually deserve different SLOs. Engines that support request priorities can serve an interactive route at a tight SLO and backfill with batch work, which raises utilisation without hurting the routes users see. If the routes cannot share a scheduler sensibly, separate replicas with different settings are simpler than one compromise configuration.

Where the ideas came from#

Yu et al.'s Orca (OSDI 2022) introduced iteration-level scheduling, letting requests join and leave the batch at every step instead of waiting for a whole batch to finish. Kwon et al.'s vLLM (2023) made large dynamic batches affordable by paging the KV cache (see PagedAttention). Agrawal et al.'s Sarathi-Serve (OSDI 2024) split prefills into near-equal chunks and built stall-free schedules that admit new requests without pausing ongoing decodes, reporting 2.6x higher serving capacity than vLLM for Mistral-7B on one A100 under tail-latency constraints (see chunked prefill). Zhong et al.'s DistServe (OSDI 2024) framed the objective as goodput under separate TTFT and TPOT constraints and removed prefill-decode interference entirely by running the phases on different GPUs, reporting 7.4x more requests or 12.6x tighter SLOs with over 90% attainment; Patel et al.'s Splitwise (2023) arrived at a similar phase split (see prefill-decode disaggregation).

Worked example: what one GPU can promise#

Take an 8B dense model in bf16 on one H100-class GPU, with assumed constants: 16 GB of weights, $\beta =3.35$ TB/s, ${\pi }_{\text{eff}}=593$ TFLOP/s (60% of the dense bf16 peak), $c_0=1$ ms, $\kappa =131,072$ bytes per token (32 layers, 8 KV heads of 128, bf16). Traffic: Poisson arrivals, prompts lognormal with median 800 tokens (mean about 1,100), outputs lognormal with median 200 (mean about 275). SLOs: TTFT at most 0.5 s and TPOT at most 25 ms, for 90% of requests.

Decode floor. A decode step with 110 sequences of about 940 tokens of context reads $16+110\times 940\times 131072/10^9\approx 29.5$ GB, which takes 8.8 ms, plus $c_0$: about 9.8 ms per step. Prefill load. At 20 requests per second, prompts add $20\times 1100=22,000$ tokens per second at $2P=16$ GFLOP each, 352 TFLOP/s, or 59% of the effective compute. Prefill, not decode, is what fills this GPU. Concurrency. By Little's law, at 20 requests per second, 275 output tokens and roughly 15 ms per token, about $20\times 275\times 0.015\approx 83$ sequences are decoding at any time, more at the tail, so a cap of 64 sequences will queue requests before compute runs out. Chunk size. A 2,048-token prefill chunk costs $2048\times 16\text{GFLOP}/593\text{TFLOP/s}=55$ ms of compute, so any decoding request sharing that step blows its 25 ms TPOT budget; a 512-token chunk costs 13.8 ms, which fits.

Simulated: the knobs, swept#

We wrote a discrete-event simulator of a continuous-batching engine in numpy (seeded, 2,500 requests per point, the first 200 dropped as warm-up) with exactly the step-time model, constants and traffic above. Each step schedules one token for every decoding sequence, then fills the remaining token budget with prefill chunks first-come-first-served; a request is admitted when there is a free sequence slot and KV room for its prompt plus its output. This is a model of a scheduler, not a measurement of any engine, but it reproduces the qualitative behaviours reported in the papers above, and every number below comes from running it.

The sequence cap is the first-order knob. At 16 concurrent sequences goodput was 5 to 6 requests per second, because requests queued for slots while the GPU had spare bandwidth: TPOT p99 stayed at 10 ms but TTFT exploded (median 44 s at 10 requests per second). Goodput rose to 17 to 19 at 64 and 22 at 128, then stopped, because past 128 the budget, not the cap, was binding. This is Little's law: below the concurrency the traffic needs, the cap creates a queue.

The token budget trades TTFT against TPOT. At 128 sequences and 20 requests per second, a 256-token budget kept TPOT p99 at 9 ms but stretched prefills over many steps, pushing TTFT p99 to 1.55 s (76% attainment). A 2,048-token budget did the opposite: TTFT was short when the queue was short, but TPOT p99 rose to 30 ms, over the SLO, because decoding requests shared steps with 55 ms prefill chunks (50% attainment). At 512 tokens both fit (TTFT p99 0.58 s, TPOT p99 14.8 ms, 97.6% attainment). With chunking disabled, so that whole prompts of up to 8,000 tokens run in one step, attainment at 20 requests per second was 15%.

The cliff is sharp. For 128 sequences with a 512 budget, attainment was 100% up to 18 requests per second, 97.6% at 20, 91.3% at 22, 44% at 24 and 11% at 26. Average latency gives no warning: TPOT p99 was flat at 14.8 ms across that whole range while TTFT went from 0.4 to 5.6 s. Operate with headroom below the knee, and alert on queue length rather than on mean latency.

The right budget depends on the SLO. With looser targets (TTFT 2 s, TPOT 50 ms), the 2,048-token budget at 256 sequences reached 24 requests per second, the same as 512, and an 8,192 budget reached 23. A tight TPOT target is what makes small chunks necessary.

max_tokens is a product decision that looks like a tuning knob. Capping outputs at 256 tokens instead of 1,024 cut the token work and kept 88.6% attainment at 20 requests per second where the 1,024 cap had 50% (at 128 sequences, budget 2,048), but it did so by truncating every answer longer than 256 tokens. Use it to bound pathological generations, set from the output-length distribution of the model you actually serve, not to buy goodput.

Post-trained models move the operating point#

The traffic parameters in the example were assumptions; for a fine-tuned model they are outcomes of post-training. Output length. Preference tuning can make a model more verbose (length bias is a known failure of DPO-style objectives, which SimPO and others correct for), and RL with verifiable rewards often lengthens reasoning traces. In our simulator, doubling the median output from 200 to 400 tokens cut goodput from 21 to 14 requests per second at 128 sequences (17 at 256, because more concurrency is needed), and 800 tokens cut it to 7. Re-measure the output-length distribution after every post-training run and re-tune. Prompt length. A fine-tune that relies on a long system prompt or few-shot examples raises prefill load on every request; prefix caching can remove most of it if the prefix is shared, and distilling the prompt into the weights removes it entirely. Adapters. Serving through multi-LoRA adds per-step cost that grows with distinct adapters in the batch, shifting the TPOT floor. Quantization lowers the decode floor (INT4 weights) or the prefill cost (FP8), so the binding constraint, and the best chunk size, can change. Structured output and speculation. Constrained decoding adds per-token CPU work that matters at small TPOT budgets, and speculative decoding lowers TPOT at low load but its benefit depends on draft acceptance rates, which fall if the draft was not trained on the fine-tune's distribution.

A tuning procedure#

1. Write the SLO down: TTFT and TPOT thresholds, attainment target (e.g. 90% or 99%), per route.
2. Capture real traffic for the post-trained model: arrival process, prompt and output length
   distributions (after the fine-tune; not the base model's), shared prefixes, adapter mix.
3. Fix hardware layout first (TP degree, replicas; see parallelism-for-serving).
4. Set max_num_seqs from Little's law: >= 1.5 * rate * mean_output_tokens * target_TPOT,
   bounded by KV-cache memory.
5. Set the per-step token budget so the worst step fits the TPOT target:
     c0 + 2 * P * budget / pi_eff  <=  TPOT target (with margin for decode tokens).
6. Load-test with replayed traffic at increasing rate; record TTFT/TPOT percentiles and attainment.
7. Goodput = highest rate with attainment >= target. Sweep budget and max_num_seqs around the
   starting point; keep the best.
8. Operate at 70 to 85% of goodput; autoscale on queue depth and TTFT percentile, not on mean latency.
9. Repeat steps 2 to 7 after any post-training run, quantization change or prompt change.

The knobs and their effects#

When each lever helps#

Failure modes and how to detect them#

Tuning on mean latency. Means stay flat until the queue becomes unstable; in our simulation TPOT p99 was constant while attainment fell from 91% to 11% over 4 requests per second. Track percentiles and attainment against the SLO. Tuning on synthetic lengths. Fixed-length benchmarks (say, 512 in and 128 out) miss the tail prompts that cause TPOT spikes and the tail outputs that hold slots; replay real length distributions. Tuning on the base model. A fine-tune's output lengths can differ by a factor of two or more; our simulator turned that into a 1.5 to 3x change in goodput. Preemption storms. When KV memory runs out, engines preempt and recompute or swap sequences, which shows up as bursts of TTFT and TPOT outliers and a rising preemption counter; lower max_num_seqs or add memory. Head-of-line blocking by very long prompts without chunking. Overload without admission control: past the knee every request misses its SLO, which is worse than rejecting some; shed or queue at the gateway. Coordinated omission in load generators that wait for responses before sending more, hiding queueing delay; use open-loop arrivals.

Implementation sketch#

The settings map directly onto engine flags; the simulator below is a stripped-down version of the one used for the numbers above and is enough to find the knee before a real load test. Run as shown, it printed an attainment of 0.976 at 20 requests per second with 128 sequences and a 512-token budget, the same as the full simulator. In POFE, a serving profile records the SLO, the traffic sample and the chosen settings next to each deployed fine-tune, so a new post-training run can be re-tuned against the same targets.

# vllm serve MODEL --max-num-seqs 128 --max-num-batched-tokens 512 \
#     --enable-chunked-prefill --gpu-memory-utilization 0.90
import numpy as np

def step_time(n_tok, ctx, P=8e9, wb=16e9, bw=3.35e12, pi=593e12, kappa=131072, c0=1e-3):
    return c0 + max((wb + kappa * ctx) / bw, 2 * P * n_tok / pi)

def simulate(rate, max_seqs, budget, n=2500, seed=0):
    r = np.random.default_rng(seed)
    arr = np.cumsum(r.exponential(1 / rate, n))
    pl = np.clip(r.lognormal(np.log(800), 0.8, n).astype(int), 16, 8000)
    ol = np.clip(r.lognormal(np.log(200), 0.8, n).astype(int), 1, 1024)
    t, i, wait, run = 0.0, 0, [], []
    first, done = np.full(n, np.nan), np.full(n, np.nan)
    while i < n or wait or run:
        while i < n and arr[i] <= t: wait.append(i); i += 1
        if not wait and not run: t = arr[i]; continue
        while wait and len(run) < max_seqs: run.append([wait.pop(0), 0, 0, 0])   # [id, prefill_done, ctx, generated]; no KV admission check here
        dec = [q for q in run if q[1] >= pl[q[0]]]; pre = [q for q in run if q[1] < pl[q[0]]]
        n_tok, ctx, left, chunks = len(dec), sum(q[2] for q in dec), budget - len(dec), []
        for q in pre:
            if left <= 0: break
            c = min(pl[q[0]] - q[1], left); chunks.append((q, c)); left -= c; n_tok += c
        t += step_time(n_tok, ctx)
        for q in dec: q[2] += 1; q[3] += 1
        for q, c in chunks:
            q[1] += c; q[2] += c
            if q[1] >= pl[q[0]]: q[3] = 1; first[q[0]] = t
        for q in [q for q in run if q[1] >= pl[q[0]] and q[3] >= ol[q[0]]]: done[q[0]] = t; run.remove(q)
    ttft = first - arr; tpot = (done - first) / np.maximum(ol - 1, 1)
    return ttft[200:], tpot[200:]

ttft, tpot = simulate(20, 128, 512)
print("attainment", np.mean((ttft <= 0.5) & (tpot <= 0.025)))

Further reading#

Zhong et al. (2024, DistServe) for the goodput definition and phase disaggregation; Agrawal et al. (2024, Sarathi-Serve) for chunked prefill and stall-free scheduling; Yu et al. (2022, Orca) for iteration-level scheduling; Kwon et al. (2023) for paged KV memory; Patel et al. (2023, Splitwise) for phase splitting across machine types; Little (1961) for the queueing law behind the sequence cap. For what these settings cost, continue to cost per million tokens and utilization and batching economics.

Frequently asked questions#

References#

1. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv.org/abs/2401.09670
2. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arxiv.org/abs/2403.02310
3. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. www.usenix.org/conference/osdi22/presentation/yu
4. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arxiv.org/abs/2309.06180
5. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, et al. (2023). Splitwise: Efficient generative LLM inference using phase splitting. arxiv.org/abs/2311.18677
6. John D. C. Little (1961). A Proof for the Queuing Formula: L = λW. doi.org/10.1287/opre.9.3.383

Related reading