Utilisation and batching economics: why idle GPUs dominate the cost of a token

Utilisation is the share of the GPU time you pay for that does useful work, and for most self-hosted language-model deployments it is the largest single number in the cost of a token. A GPU costs the same per hour whether it decodes 116 sequences at once or one, and whether it serves the lunchtime peak or the 4 a.m. trough. Batching and utilisation are two views of the same quantity: batching is how much work a GPU does per step while it is running, utilisation is how much of the paid time it is running at a useful batch. This article derives why idle and under-batched capacity dominates cost, computes the break-even utilisation against any per-token price, simulates a week of diurnal traffic under four provisioning policies, and puts a number on the value of filling troughs with training and batch work.

Why idle GPUs dominate cost#

Start from the unit-cost formula in cost per million tokens: a GPU priced at $h$ per hour that serves $R$ tokens per second while busy and is busy a fraction $u$ of the time costs

The cost is hyperbolic in $u$. Going from 80% to 40% doubles it; going from 40% to 10% quadruples it again. Hardware and kernel work typically changes $R$ by tens of percent, while utilisation across real fleets varies by factors of several. Barroso and Hölzle (2007) observed that servers operate most of the time at between 10 and 50 percent of their maximum utilisation. Jeon et al. (2019) analysed a large multi-tenant GPU cluster for deep-learning training and identified gang scheduling and locality constraints as recurring sources of lost GPU utilisation. Hu et al. (2024) characterised an LLM development data centre and found median GPU utilisation of 97% and 99% in its two clusters by the device-level counter, yet median streaming-multiprocessor activity of around 40%. The second number is the warning: "the GPU is busy" and "the GPU is doing useful work at a good batch size" are different measurements, and only the second one buys tokens.

With continuous batching, traffic sets the batch#

A server with continuous batching (Orca's iteration-level scheduling, Yu et al., 2022; vLLM's paged KV cache, Kwon et al., 2023) admits new requests into the running batch at every decode step. You configure a maximum batch, but the batch it actually runs is set by traffic. Little's law (Little, 1961) says the average number of requests in a system equals arrival rate times time in the system. For decode, the requests in the system are the batch $b$, the arrival rate is ${\lambda }_{\text{req}}$ requests per second, and each request spends $n_{\text{out}}$ steps of length $t_{\text{step}}(b)$ decoding:

Given the served token rate ${\lambda }_{\text{tok}}$, this fixes $b$, and therefore the step time each user sees. An always-on GPU's cost per token is then simply $h\times 10^6/(3600{\lambda }_{\text{tok}})$: every token pays its share of a full GPU-hour, however few tokens there are. In this view the "utilisation" $u$ is ${\lambda }_{\text{tok}}/R$, the served rate as a share of the rate at the batch your latency target allows.

We solved this fixed point for the 8B-parameter model of the cost-per-million article (one 80 GB GPU, 1,500 prompt and 300 completion tokens, prefill time-sharing the GPU, $2.50 per GPU-hour assumed).

At 200 output tokens per second the GPU runs a batch of 2, each user sees 10 ms per token, and the cost is $3.47 per million output tokens. At 1,000 tokens per second, batch 13 and 13 ms, $0.69. At 2,000, batch 41 and 20 ms, $0.35. At 2,920 tokens per second the batch reaches 116 and the step time reaches a 40 ms per-token target; this is the capacity $R$ we use for sizing, and the cost there is $0.24. The GPU could push on to 3,320 tokens per second before memory runs out, but only by breaking the latency target. A lightly loaded GPU is not just idle part of the time; it runs small, inefficient batches all of the time, and that shows up in cost exactly like idleness.

Break-even utilisation against a per-token price#

If the alternative is a per-token price $p$ per million output tokens (a hosted open-model API, or an internal chargeback rate), self-serving is cheaper only when $c(u)<p$:

Worked by hand with $h=\$2.50$, $R=2,920$ and an assumed $p=\$0.40$: $u^{\ast }=2.5\times 10^6/(3600\times 2920\times 0.40)=0.595$. The GPU must average 59% of its latency-constrained capacity, around the clock, to match that price on GPU cost alone, before any engineering or redundancy cost. Against $p=\$1$, $u^{\ast }=24\%$; against $p=\$2$, 12%. Against $p=\$0.20$, $u^{\ast }$ is 119%: no utilisation can match it, which is what a large shared service with better batching and pooled traffic can do to a single-tenant deployment.

A simulated week of traffic#

Setup

We generated a week of traffic at five-minute resolution (seed 42): a daily cycle peaking in the afternoon with a trough near zero overnight, weekends at 55% of weekdays, and 8% log-normal noise per interval. Mean demand is 9,000 output tokens per second (5.44 billion output tokens a week); the peak is 28,463, a peak-to-average ratio of 3.16. Each GPU serves $R=2,920$ output tokens per second at the 40 ms target, and every policy keeps 25% headroom ($\theta =0.75$). The four policies, with labelled price assumptions:

Results

The static fleet's utilisation is exactly what the sizing rule predicts: with peak-to-average 3.16 and 75% headroom, average utilisation cannot exceed $0.75/3.16=23.7\%$. The serving work in every policy costs about $0.18 to $0.24 per million tokens; everything else is paying for capacity that is not serving. Autoscaling more than halves the GPU-hours and, despite a 32% higher hourly price, cuts cost per token by 47%. Committing the always-on floor at the reserved price saves another 9%. Filling the static fleet's troughs with preemptible work is the cheapest option of all at $0.355, but only if that work exists and is worth the committed price; if it is worth half, the net cost is $0.56.

The static bill is easy to check by hand. Thirteen GPUs for 168 hours is 2,184 GPU-hours, which at $1.90 is $4,150 a week. Demand averaged 9,000 output tokens per second, which at 2,920 per GPU is the work of 3.08 GPUs, or 518 GPU-hours a week. The other 1,666 GPU-hours, $3,166 or 76% of the bill, bought capacity that waited for the Friday peak. Divide $4,150 by the week's 5.44 billion output tokens and you get the $0.762 in the table; divide only the busy share and you get the $0.181 that a benchmark-based estimate would have predicted. The factor of 4.2 between them is the peak-to-average ratio divided by the headroom fraction, $\pi /\theta =3.16/0.75$, and no kernel optimisation touches it.

The same arithmetic explains why the hourly price matters less than the number of hours. The autoscaled fleet pays 32% more per hour but buys 60% fewer hours. A procurement discussion that focuses on the hourly rate while leaving provisioning static is optimising the smaller factor.

The autoscaler never ran short in this simulation because 25% headroom absorbed the demand growth during a 10-minute cold start. Shorten the headroom or lengthen the cold start (large models on slow storage can take several minutes to load) and the autoscaled policy starts missing its latency target at the morning ramp. Cold-start time is therefore an economic parameter: it sets how much headroom autoscaling needs, and headroom is idle capacity.

The levers, ranked by what they move#

Pooling workloads

The cheapest utilisation gain is statistical: put several workloads on one fleet. Using the same daily shape, four identical workloads in one time zone pooled have a peak-to-average of 2.90 (noise averages out), which raises the static utilisation cap from 23% to 26%. Four workloads in time zones three hours apart have 2.20 (cap 34%). Three workloads eight hours apart have 1.41 (cap 53%). Pooling is why a shared per-token service can price below what a single tenant can reach, and it is the strongest argument for serving many fine-tuned variants as adapters on one base model: S-LoRA (Sheng et al., 2023) and Punica (Chen et al., 2023) make that practical, and multi-LoRA serving covers the mechanics.

Autoscaling and cold starts

Autoscaling converts a peak-sized fleet into a demand-shaped one. Its cost is cold-start time and the minimum replica count. Reduce cold start by keeping weights on fast local storage, loading adapters rather than full models, and keeping one warm spare instead of headroom on every replica.

Larger effective batch at the same latency

Raising $R$ at fixed latency target lowers $c$ at every utilisation level. The relevant techniques are those that let more sequences share a step within the same time per token: paged KV memory, KV-cache compression and lower-precision caches, chunked prefill to stop long prompts stalling decodes, and prefill-decode disaggregation at larger scale. Sarathi-Serve (Agrawal et al., 2024) reported 2.6× higher serving capacity under tail-latency constraints for a 7B-class model on one GPU from chunked prefills and stall-free batching. These raise the ceiling; they do nothing for a fleet whose problem is a low floor.

Off-peak training and batch work

The static fleet above leaves 1,666 GPU-hours idle each week, 1,166 of which we assumed could be filled. As a scale reference, at 40% FLOP utilisation and the $6P$ FLOPs-per-token rule of thumb for full-parameter training, 1,166 GPU-hours is the compute for about 35 billion training tokens of an 8B model, or hundreds of typical fine-tuning runs. In practice the best fillers are work that tolerates preemption and does not need many GPUs at once: offline evaluation, synthetic-data generation, reward-model scoring, rollouts for GRPO-style RL, and rejection-sampling data collection, all of which are inference workloads that batch well. Training jobs need checkpointing frequent enough that a preemption at the morning ramp costs minutes, not hours. SpotServe (Miao et al., 2023) addresses the mirror image, serving on preemptible capacity, and reported 54% savings against on-demand instances. Hu et al. (2024) give a cautionary detail for evaluation jobs: in their HumanEval runs about half of the wall time went to model loading, preprocessing and correctness tests rather than GPU inference, so trough-filling jobs must be packaged to start fast.

Failure modes and how to detect them#

Measuring the wrong utilisation. The device counter reports that a kernel was running, not that the batch was large. Track served tokens per paid GPU-hour per model, and the running batch size distribution from the serving engine. A fleet with 95% "GPU utilisation" and a median running batch of 3 is the expensive point on the load curve.

Sizing every replica for peak. If utilisation over a week is close to $\theta /\pi$, the fleet is static in all but name. Plot replica count against demand for a week; a flat line means autoscaling is off or pinned at its maximum.

Too many small deployments. Each model with its own redundancy floor has its own idle capacity. Count deployments with average utilisation under 10%; they are usually candidates for consolidation onto a shared base with adapters.

Trough-filling that preempts badly. If batch jobs do not yield quickly, the morning ramp collides with them and serving latency degrades. Watch time-to-first-token at the ramp hour specifically, and alert on preemption latency.

Chasing throughput on an idle fleet. A 30% faster kernel moves the serving segment of each bar, which is the minority. Before optimising $R$, compute $\theta /\pi$ and the deployment count.

What to do at each utilisation level#

Implementation sketch#

The simulation needs only a demand series and a sizing rule. This version reproduces the static and autoscaled policies; plug in your own five-minute demand export.

import math, numpy as np

def fleet_cost(demand, R=2920, headroom=0.75, policy="autoscale", min_rep=2,
               cold_start_slots=2, down_after=6, h_reserved=1.90, h_on_demand=2.50, slot_h=1/12):
    """demand: output tokens/s per 5-minute slot. Returns $/1M output tokens, utilisation, misses."""
    if policy == "static":
        g = np.full(len(demand), math.ceil(demand.max() / (headroom * R)))
        price = h_reserved
    else:
        g, cur, pending, low = np.zeros(len(demand)), min_rep, [], 0
        for i, lam in enumerate(demand):
            cur += sum(n for due, n in pending if due <= i)
            pending = [(d, n) for d, n in pending if d > i]
            want = max(min_rep, math.ceil(lam / (headroom * R)))
            if want > cur + sum(n for _, n in pending):
                pending.append((i + cold_start_slots, want - cur - sum(n for _, n in pending)))
            low = low + 1 if want < cur else 0
            if low >= down_after:
                cur, low = want, 0
            g[i] = cur
        price = h_on_demand
    tokens = demand.sum() * slot_h * 3600
    cost = g.sum() * slot_h * price
    return cost / tokens * 1e6, demand.sum() / (g * R).sum(), int((demand > g * R).sum())

Run it on last month's traffic before buying capacity. In POFE, serving many fine-tuned adapters on one base deployment is the default for exactly the pooling reason above.

Further reading#

Barroso and Hölzle (2007) is the short classic on why servers run far below peak. Jeon et al. (2019) and Hu et al. (2024) are measurement studies of GPU clusters, the second specific to LLM development and the clearest evidence that device counters overstate useful work. Little (1961) is the proof behind the batch-from-traffic argument. Yu et al. (2022), Kwon et al. (2023) and Agrawal et al. (2024) are the serving systems that make large batches achievable under latency targets. Sheng et al. (2023) and Chen et al. (2023) cover multi-adapter serving, the main pooling tool for fine-tuned models, and Miao et al. (2023) covers preemptible capacity. For the full cost picture including engineering and redundancy, see self-hosting TCO.

Frequently asked questions#

References#

1. Luiz André Barroso, Urs Hölzle (2007). The Case for Energy-Proportional Computing. doi.org/10.1109/MC.2007.443
2. Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, Fan Yang (2019). Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. www.usenix.org/conference/atc19/presentation/jeon
3. Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, et al. (2024). Characterization of Large Language Model Development in the Datacenter. arxiv.org/abs/2403.07648
4. John D. C. Little (1961). A Proof for the Queuing Formula: L = λW. doi.org/10.1287/opre.9.3.383
5. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. www.usenix.org/conference/osdi22/presentation/yu
6. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arxiv.org/abs/2309.06180
7. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arxiv.org/abs/2403.02310
8. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arxiv.org/abs/2311.03285
9. Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy (2023). Punica: Multi-Tenant LoRA Serving. arxiv.org/abs/2310.18547
10. Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, Zhihao Jia (2023). SpotServe: Serving Generative Large Language Models on Preemptible Instances. arxiv.org/abs/2311.15566

Related reading