Cost per million tokens, from first principles

Cost per million tokens is the unit price of language model inference: the money you spend to read (prefill) or write (decode) one million tokens. Every price list quotes it, but on hardware you control it is not a given. It falls out of five quantities: what a GPU costs per hour, how many tokens per second that GPU produces at the batch size you actually run, what share of the paid hours do useful work, and how your traffic splits between input and output tokens. This article derives the formula from first principles, builds a small roofline-based model of an 8B-parameter dense decoder on one 80 GB data-centre GPU, and shows which inputs move the answer. It is the base unit for the rest of the economics series: cost per task, self-hosting TCO and the ROI of a fine-tune all start from it.

Why derive it instead of reading it off a benchmark#

A throughput benchmark answers one question: how many tokens per second did this engine produce for this model, at this batch size, with these prompt and completion lengths. Change any of those and the number changes, often by an order of magnitude. A post-trained model makes this worse, not better. A fine-tune that shortens the system prompt from 2,000 tokens to 200 changes the prefill bill. A LoRA adapter served unmerged adds a small matmul per layer. A quantised fine-tune changes the bytes moved per decode step. A domain tokenizer changes how many tokens the same text costs. You need a model that tells you how each of these propagates into dollars.

The ingredients have been public for a while. Pope et al. (2022) wrote down an analytical model of transformer inference cost in terms of memory bandwidth, FLOPs and batch size, and showed that decode is bound by memory traffic at small batch while large-batch prefill reached 76% model FLOPs utilisation on their hardware. Yu et al. (2022) introduced iteration-level scheduling in Orca, which is what lets a server keep a large decode batch full as requests arrive and finish; they reported 36.9× higher throughput than FasterTransformer, which batched whole requests, at the same latency. Kwon et al. (2023) made large batches fit in memory with paged KV-cache management in vLLM, reporting 2 to 4× throughput over prior systems. Yuan et al. (2024) survey the roofline view of LLM inference. Griggs et al. (2024) showed that the cheapest GPU type depends on request size, request rate and latency target, and that mixing GPU types cut deployment cost by up to 77% in a conversational setting. The model below compresses this literature into a calculator you can reason about by hand.

The formula, derived#

From GPU-hours to tokens

Let $h$ be the price of one GPU-hour, $R$ the number of tokens per second the GPU produces while busy, and $u\in (0,1]$ the utilisation: the share of paid GPU time that is spent producing tokens you sell or use. In one paid hour you get $3600uR$ tokens, so

Everything interesting hides in $R$, and $R$ differs by phase. Prefill processes all input tokens of a request in parallel and is limited by arithmetic. Decode produces one token per sequence per forward pass and is limited by how many bytes must be read from GPU memory per pass. So we need two rates, $R_{\text{pre}}$ and $R_{\text{dec}}$, and two prices, $c_{\text{in}}$ and $c_{\text{out}}$.

Decode: one step serves the whole batch

Take a dense decoder with $P$ parameters stored at ${\beta }_w$ bytes each, $n_{\ell }$ layers, $n_{kv}$ key-value heads of dimension $d_h$, model width $d$, and KV cache stored at ${\beta }_{kv}$ bytes per value. Each token held in the KV cache costs

(the 2 is one key and one value). One decode step for a batch of $b$ sequences with average live context $\bar{L}$ must read all weights once and every sequence's cache once, and it must do about $2P$ FLOPs of matmul plus $4n_{\ell }d\bar{L}$ FLOPs of attention per sequence. On a GPU with peak memory bandwidth $B$ and peak dense throughput $F$, with achieved fractions $\eta$ (bandwidth) and $m$ (FLOPs), the step time is the larger of the two bounds plus a fixed per-step overhead ${\tau }_0$ for scheduling, sampling and kernel launches:

This is the roofline argument from the inference roofline applied to money. At $b=1$ the weight read $P{\beta }_w$ dominates and every token pays for all of it. As $b$ grows the weight read is shared across $b$ tokens, so cost per token falls almost as $1/b$, until either the KV term $b\bar{L}\kappa$ dominates the memory traffic or the batch runs out of memory. The memory ceiling, with 10% of HBM $M$ held back for activations and fragmentation, is

where $L_{\max }$ is the context each sequence may grow to. Paged KV management (see PagedAttention) is what makes the real batch approach this bound instead of a much smaller one set by fragmentation.

Prefill: arithmetic-bound and cheap per token

Prefill of an $L_{\text{in}}$-token prompt does about $2P+2n_{\ell }d{L}_{\text{in}}$ FLOPs per token (the attention term averages over causal positions), and with enough tokens in flight it is compute-bound:

Per request and blended

A request with $n_{\text{in}}$ input and $n_{\text{out}}$ output tokens occupies $n_{\text{in}}/R_{\text{pre}}+n_{\text{out}}/R_{\text{dec}}$ busy GPU-seconds, so

The additive form assumes prefill and decode time-share the same GPU, which is what continuous batching with chunked prefill does. With prefill-decode disaggregation the two phases run on separate pools and you price each pool with its own $h$, $u$ and $R$; the formula keeps its shape. For a tensor-parallel deployment over $g$ GPUs, replace $h$ by $gh$ and use the group's aggregate $R$. For a mixture-of-experts model, the FLOPs term uses active parameters, while the memory term uses whatever expert weights a step actually touches, which at large batch approaches all of them.

The assumptions, stated#

Every number in this table is an assumption chosen to be plausible, not a measurement of any product. Substitute your own.

Worked example by hand#

Take a batch of $b=64$ sequences with average live context $\bar{L}=1,650$ tokens (1,500 prompt tokens plus half of a 300-token completion). The KV cache in flight is $64\times 1,650\times 131,072=13.84$ GB. Add 16 GB of weights and one decode step reads 29.84 GB. At 70% of 3.35 TB/s, that is $29.84/2.345=12.73$ ms. The arithmetic is $64\times (1.6\times 10^{10}+4\times 32\times 4096\times 1650)\approx 1.08\times 10^{12}$ FLOPs, which at half of 989 TFLOP/s takes 2.18 ms, so the step is memory-bound by a factor of almost six. With 3 ms of overhead the step takes 15.73 ms and the GPU emits $64/0.01573=4,070$ tokens per second. At $2.50 an hour and 50% utilisation:

Prefill runs at $0.5\times 989\times 10^{12}/(1.6\times 10^{10}+2\times 32\times 4096\times 1500)=30,165$ tokens per second, so $c_{\text{in}}=\$0.046$ per million input tokens. Output is 7.4 times as expensive as input at this batch size. A chat request of 1,500 in and 300 out costs $0.0694+0.1025=0.17$ thousandths of a dollar, about $0.17 per thousand requests.

If you instead push the batch to the memory ceiling for 1,800-token sequences, $b_{\max }=\lfloor (72-16)\text{GB}/(1800\times 131,072)\rfloor =237$. Decode rises to 7,481 tokens per second, output cost drops to $0.186 per million, input stays at $0.046, and the output to input price ratio is 4.0. That ratio is not a pricing convention. It falls out of the physics: one decode token costs a share of a memory-bound step, one prefill token costs a share of a compute-bound pass.

Batch size is the biggest lever you control#

At batch 1 every configuration costs about $14 per million output tokens: the GPU streams 16 GB of weights to produce one token. Batch 16 brings that to $0.98, batch 64 to $0.34. The curve is close to $1/b$ while weights dominate the bytes moved, and flattens once the KV cache does. Context length decides where it flattens and where it stops. With 1k-token prompts the batch can reach 328 and output costs $0.13 per million. With 4k prompts the ceiling is 99 sequences and $0.46. With 16k prompts only 26 sequences fit and the floor is $1.78, more than thirteen times the short-context figure. Long context is expensive not because attention FLOPs are large but because each sequence's cache takes memory away from the batch.

Batch size is not free. Every sequence in the batch waits for the same step, so time per output token rises with $b$.

Under a 40 ms per-token target the memory ceiling binds first for all three prompt lengths. Tighten the target and the latency objective takes over: at 30 ms the 1,800-token workload can batch 218 sequences ($0.191 per million output tokens), and at 20 ms only 110 ($0.252). That 36% cost increase for a stream about 1.6× faster (20 ms against 31.7 ms per token) is the trade that latency SLO tuning is about. DistServe (Zhong et al., 2024) frames the same idea as goodput, the request rate a GPU can serve while meeting both time-to-first-token and time-per-token targets, and reports 7.4× more requests within SLO by splitting the phases.

The input/output mix decides which phase you are paying for#

Per-token prices hide the shape of a workload. The same GPU-hour buys very different numbers of requests depending on how tokens split between prompt and completion, and long prompts also shrink the feasible decode batch.

Four lessons come out of the bars. Retrieval-heavy workloads are prefill workloads: at 8,000 prompt tokens, $0.407 of the $0.677 per thousand requests is prefill, and output costs $0.90 per million because only 51 sequences fit. Prefix caching attacks exactly this bar when prompts share a long system prefix. Extraction (3,000 in, 60 out) is almost entirely prefill, which is why a fine-tune that removes few-shot examples from the prompt cuts its cost nearly in proportion. Drafting (200 in, 1,500 out) is a decode workload, and its levers are batch size, speculative decoding and lower-precision weights and cache. Chat sits between, and its blended cost of $0.069 per million tokens hides an output price four times the input price.

For a workload with input-to-output ratio $\rho =n_{\text{in}}/n_{\text{out}}$, the blended price per million tokens is $(\rho c_{\text{in}}+c_{\text{out}})/(\rho +1)$, which is how to compare a self-hosted deployment with an API that bills the two separately. Comparing on output price alone overstates self-hosting for retrieval workloads and understates it for drafting.

Which assumptions move the answer#

We varied each input across its range with everything else at base, for the chat mix at the memory-ceiling batch.

Utilisation and GPU price dominate, and they enter the formula identically: halving utilisation doubles cost, so a cheap GPU that sits idle is more expensive than a dear one that is busy. Going from 50% to 25% utilisation moves the blended price from $0.069 to $0.139; going to 80% brings it to $0.043. Prefill efficiency comes third for this prompt-heavy mix. Bandwidth efficiency and step overhead matter less than intuition suggests at the memory ceiling, because the batch is already large. Weight quantisation to 8 bits barely moves this point (from $0.069 to $0.065) for the same reason: at batch 237, the KV cache, not the 16 GB of weights, is most of each step's traffic. Quantising the KV cache to 8 bits matters more ($0.054) because it doubles the batch the GPU can hold. With both at 8 bits the ceiling rises to 542 sequences and output cost to $0.080 per million, provided your latency target allows the 31 ms step that results. See FP8 and FP4 inference and KV-cache compression for what these cost in quality.

To see the combined uncertainty we drew 20,000 joint samples (seed 7) with GPU price, utilisation, bandwidth efficiency, step overhead and prefill efficiency each uniform over its range. The blended cost had a 10th, 50th and 90th percentile of $0.044, $0.079 and $0.145 per million tokens, and output cost alone $0.115, $0.207 and $0.382. A factor of three between optimistic and pessimistic is the honest error bar for a paper estimate; a one-day load test on your own traffic shape shrinks it more than any further modelling.

Comparing with per-token prices#

The model gives the cost of capacity you own while it is busy. To compare with a per-token price $p$ you need the utilisation at which the two are equal, which follows directly from the first formula:

As an illustration, if a hosted open-model API charged an assumed $0.20 per million output tokens for a model of this size, the dedicated GPU at batch 237 ($2.50/h, $R=7,481$) breaks even at $u^{\ast }=2.5\times 10^6/(3600\times 7481\times 0.20)=46\%$. Against an assumed frontier API output price in the $5 to $25 per million range, $u^{\ast }$ is below 2%, but the comparison is meaningless unless the 8B model does the task as well; that is the subject of the quality-adjusted comparison. Utilisation in practice is the subject of utilisation and batching economics, and everything this formula leaves out (engineering time, redundancy, compliance) is in the TCO article.

What changes for a post-trained model#

Post-training moves the inputs of this formula in ways that are easy to miss.

The adapter row deserves emphasis. S-LoRA (Sheng et al., 2023) and Punica (Chen et al., 2023) serve many LoRA adapters over one copy of the base weights; Punica reported 12× higher throughput than serving the models separately at about 2 ms added latency per token. Economically, that turns many low-utilisation deployments into one high-utilisation one, which is worth more than any kernel optimisation. See multi-LoRA serving and merging versus serving adapters.

How estimates go wrong#

Using peak or benchmark throughput

A benchmark run at a fixed concurrency of 256 tells you $R$ at $b=256$. If your traffic sustains an average of 30 concurrent sequences, your $R$ is a fraction of that, and the error passes straight into cost. Detect it by logging the running batch size from your serving engine and computing $R$ at the median observed batch, not the benchmark's.

Forgetting utilisation

Setting $u=1$ prices only busy seconds. Real services keep capacity for peaks and failures. A cost per million tokens computed without $u$ is a lower bound you will never invoice yourself at. The symptom is a monthly GPU bill that is two to four times what tokens served times model cost predicts.

Pricing only output tokens

For retrieval and extraction, prefill is most of the GPU time. A model that counts output tokens only can be off by a factor of five on a prompt-heavy workload.

Mixing units across options

Per-token, per-request and per-GPU-hour prices are only comparable after you fix the request shape. Always convert to cost per request or per task first, then compare. The cost-per-task article goes further and adds retries, tool calls and human review.

Treating "GPU utilisation" counters as $u$

The device-level utilisation counter reports whether any kernel ran in an interval, not how much useful work was done. A decode loop at batch 1 can show near 100% on that counter while delivering the $14-per-million cost from the batch chart. Use tokens served per paid GPU-hour as the ground truth.

When to use this model and what to use instead#

A calculator you can run#

The whole model fits in a few lines of Python. Replace the assumptions with your model's architecture, your GPU's datasheet, a measured $\eta$ and $m$, and your price.

def cost_per_million(P=8.0e9, n_layers=32, n_kv=8, d_head=128, d_model=4096,
                     bytes_w=2, bytes_kv=2, hbm_gb=80, bw=3.35e12, flops=989e12,
                     eta_bw=0.70, mfu=0.50, tau0=0.003, h=2.50, u=0.50,
                     n_in=1500, n_out=300, batch=None):
    kv = 2 * n_layers * n_kv * d_head * bytes_kv            # bytes per cached token
    weights = P * bytes_w
    L_max, L_avg = n_in + n_out, n_in + n_out / 2
    b_max = int((0.9 * hbm_gb * 1e9 - weights) // (L_max * kv))
    b = min(batch or b_max, b_max)
    mem = (weights + b * L_avg * kv) / (eta_bw * bw)
    comp = b * (2 * P + 4 * n_layers * d_model * L_avg) / (mfu * flops)
    r_dec = b / (tau0 + max(mem, comp))                      # output tokens/s while busy
    r_pre = mfu * flops / (2 * P + 2 * n_layers * d_model * n_in)
    usd_per_s = h / 3600 / u
    c_out, c_in = usd_per_s / r_dec * 1e6, usd_per_s / r_pre * 1e6
    per_req = usd_per_s * (n_in / r_pre + n_out / r_dec)
    blended = per_req / (n_in + n_out) * 1e6
    return dict(batch=b, r_dec=r_dec, r_pre=r_pre, c_out=c_out, c_in=c_in,
                per_1k_requests=1000 * per_req, blended=blended)

print(cost_per_million())            # batch 237: c_out ~0.186, c_in ~0.046
print(cost_per_million(batch=64))    # c_out ~0.341

To calibrate, run your serving engine (vLLM, SGLang and TensorRT-LLM all expose running batch size and token counters) at two or three concurrency levels, fit $\eta$ and ${\tau }_0$ to the measured step times, and keep the model for what-if questions. In POFE the same arithmetic sits behind the serving estimate attached to each trained model, using the model's measured prompt and completion lengths from evaluation.

Further reading#

Pope et al. (2022) is the clearest derivation of inference cost from hardware limits, including the partitioning choices for models that do not fit on one device. Yu et al. (2022, Orca) and Kwon et al. (2023, vLLM) explain the two mechanisms, iteration-level batching and paged KV memory, that make large decode batches achievable. Agrawal et al. (2024, Sarathi-Serve) and Zhong et al. (2024, DistServe) cover the throughput-latency trade at the heart of the batch chart. Patel et al. (2023, Splitwise) and Griggs et al. (2024, Mélange) treat cost directly, by splitting phases across hardware and by choosing GPU types per workload. Yuan et al. (2024) is a broad survey organised around the roofline model.

Frequently asked questions#

References#

1. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, Jeff Dean (2022). Efficiently Scaling Transformer Inference. arxiv.org/abs/2211.05102
2. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. www.usenix.org/conference/osdi22/presentation/yu
3. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arxiv.org/abs/2309.06180
4. Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arxiv.org/abs/2403.02310
5. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. arxiv.org/abs/2401.09670
6. Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, Ricardo Bianchini (2023). Splitwise: Efficient generative LLM inference using phase splitting. arxiv.org/abs/2311.18677
7. Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica (2024). Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. arxiv.org/abs/2404.14527
8. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, Ion Stoica (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arxiv.org/abs/2311.03285
9. Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy (2023). Punica: Multi-Tenant LoRA Serving. arxiv.org/abs/2310.18547
10. Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights. arxiv.org/abs/2402.16363

Related reading