Self-hosting vs API: the total cost of ownership of LLM serving

Total cost of ownership (TCO) for language-model serving is everything you pay per month to answer requests, not only the GPU bill or the token bill. For an API it is mostly tokens plus a little integration and vendor-management work. For a self-hosted model it is GPUs sized for peak traffic and failures, the engineers who keep them running, the compliance work of operating a production system that holds customer data, tooling, and the training and evaluation that keep the model current. This article builds a TCO model for four ways of serving the same workload: a frontier API, a hosted open-model API serving your fine-tuned model per token, dedicated GPUs you run yourself, and servers you own on-premises. It derives the GPU count from traffic, prices each line item as a labelled assumption, finds the break-even volumes, and shows which inputs move them. It extends the earlier POFE study of the fine-tuning versus API break-even, which modelled operations as a flat $1,500 a month; here we ask what happens when engineering time, redundancy and compliance are priced properly.

Why GPU price is the wrong place to start#

The marginal cost of a token on a busy GPU is small, and the cost per million tokens article shows how to compute it. The trouble is that a self-hosted deployment is never only busy GPUs. It has a floor set by availability requirements, an engineering cost that does not scale with tokens, and idle capacity held for peaks. Below a certain volume those fixed items are most of the bill, and the per-token efficiency that benchmarks measure does not matter at all. Barroso, Hölzle and Ranganathan's treatment of warehouse-scale computing splits cost into capital and operating parts and makes the same point for general data-centre workloads: the price of the server is one line among several, and utilisation decides what each useful unit of work costs. Sculley et al. (2015) made the corresponding point for machine-learning systems specifically: the model code is a small fraction of a production system, and the surrounding configuration, data pipelines, monitoring and serving infrastructure carry ongoing maintenance cost that is easy to underestimate.

The TCO model#

Sizing the GPU fleet from traffic

Let $V$ be monthly output tokens (in millions) and $\rho$ the ratio of input to output tokens. The average output rate is $\bar{r}=V\cdot 10^6/(730\times 3600)$ tokens per second. Traffic is not flat, so capacity must cover the peak $r_{\text{peak}}=\pi \bar{r}$, where $\pi$ is the peak-to-average ratio. Each GPU serves $R$ output tokens per second at the batch size your latency target allows, with input processing included (the effective rate from the per-request formula in the cost-per-million article), and you run it at most at a headroom fraction $\theta$ of that rate at peak. Spreading over $G$ regions or availability zones and keeping $s$ spare replicas per region for failures and rolling deploys gives

The fraction of paid capacity that sits idle is $1-\bar{r}/(gR)$. With $G=2$ and $s=1$ the minimum fleet is four replicas, however small the traffic.

Monthly cost of each option

With per-token prices $p^{\text{in}},p^{\text{out}}$ (per million), GPU-hour price $h$, engineering effort $E$ in full-time equivalents at loaded monthly cost $w$, compliance cost $K$, tooling $T$, and training plus evaluation amortised to $M$ per month:

For on-premises servers, $P_{\text{srv}}$ is the purchase price of an eight-GPU server depreciated straight-line over $L$ months, kW its power draw, PUE the facility's power usage effectiveness, $e$ the electricity price per kWh, and $c_{\text{colo}}$ space, cooling and network per server; $n_{\text{srv}}=\max (2,\lceil g/8\rceil )$. Dividing the bracket by $8\times 730$ gives the equivalent price of an owned GPU-hour at full allocation, which is the number to compare with $h$.

Break-even between two options is the smallest $V$ at which the one with the larger fixed cost is cheaper. Where both are linear, as with two APIs, it has the closed form $V^{\ast }=\Delta F/\Delta k$: the difference in fixed cost divided by the difference in cost per million output tokens. Where one side has the stepped fleet $g(V)$, we solve it numerically on a fine grid.

Assumptions#

Every value is an illustrative assumption, not a quote. The workload is an 8B-parameter fine-tuned model serving a chat-like mix with three input tokens per output token.

Compliance here means the recurring work of operating a system that processes customer data: access reviews, audit evidence, penetration tests, incident runbooks, data-retention controls. For an API it is vendor due diligence, contract review and a redaction or logging layer. Both are real and both are often left out.

Worked example by hand: 3 billion output tokens a month#

At $V=3,000$ million output tokens, the average output rate is $3\times 10^9/(730\times 3600)=1,142$ tokens per second. Peak is $2.5\times 1142=2,854$. One GPU at 80% headroom serves $0.8\times 2900=2,320$, so serving needs $\lceil 2854/2320\rceil =2$ GPUs, one per region, plus one spare in each: $g=4$. The fleet's capacity is $4\times 2900=11,600$ tokens per second against an average demand of 1,142, so 90% of paid capacity is idle.

Dedicated: GPUs $4\times 730\times 2.50=\$7,300$, engineering $\$22,000$, compliance $\$3,000$, tooling $\$1,000$, training and eval $\$4,000$: $37,300 a month. Frontier API: tokens $3000\times (3\times 3+15)=\$72,000$, plus $0.15\times 22,000=\$3,300$ and $\$1,500$: $76,800. Hosted open-model API: tokens $3000\times (3\times 0.10+0.40)=\$2,100$ plus $\$3,300+\$1,500+\$4,000$: $10,900. On-prem: two servers at $300,000/36+10\times 1.3\times 730\times 0.12+1,500=\$10,972$ each, 1.5 FTE, and the same compliance, tooling and training: $62,944.

The GPUs are 20% of the dedicated option's cost. Engineering is 59%.

The cost curves#

Under these assumptions the break-even volumes are:

The earlier research page put the break-even with a frontier API at about 427 million output tokens a month, using its own price assumptions and $1,500 a month of operations. Re-running that model with half an engineer added ($11,000 a month) moves its break-even to 1.06 billion; with one engineer ($22,000), to 1.69 billion. Engineering time alone roughly quadruples the volume at which self-hosting pays off.

The last row of the table is the result most teams do not expect. A hosted open-model API charging the assumed $0.10 and $0.40 per million costs $0.70 per million output tokens including three input tokens each. The self-run fleet's marginal cost at scale, with peak-to-average 2.5 and 80% headroom, is $2.50\times 10^6/(3600\times 2900\times 0.8/2.5)=\$0.75$ per million output tokens before any fixed cost. A shared service that pools many customers' traffic runs at higher utilisation than your single workload can, and the price reflects that. If the hosted price were twice the assumption, self-running would break even at 38.6 billion output tokens a month; at three times, 18.8 billion; at twice the price with flat batch traffic ($\pi =1.5$) in one region, 24.3 billion. These are large volumes for an 8B model.

Idle capacity is the hidden majority of the GPU bill

At a billion output tokens a month the four-GPU floor is 97% idle. At 10 billion it is 84% idle on 8 GPUs, and at 100 billion still 70% idle on 44 GPUs, because a 2.5× peak served at 80% headroom caps average utilisation at $0.8/2.5=32\%$. The utilisation and batching article covers the levers: autoscaling, filling troughs with batch and training work, and serving many adapters on one fleet.

Which assumptions move the break-even#

Engineering effort is the single largest input: half an FTE puts break-even at 896 million output tokens a month, two FTEs at 2.27 billion. The frontier API's output price is next (2.33 billion at $5, 956 million at $25), followed by the input:output ratio, because prompt-heavy workloads pay the API for many input tokens that are cheap to process yourself (834 million at $\rho =8$). GPU price, region count and compliance each move break-even by roughly 6 to 15%. Throughput, the quantity most serving engineering optimises, has no effect at all at this volume. This is the same finding as in the earlier study and it holds more strongly once engineering is priced: at the break-even point the fleet is four GPUs on a redundancy floor, and making them faster changes nothing until traffic needs a fifth.

Line items that change the answer#

Engineering time

Running a model in production is on-call rotations, serving-engine upgrades, driver and kernel compatibility, capacity planning, incident response and evaluation of each new checkpoint. One engineer's time spread across these is a conservative estimate for a single production model with an availability target; a team serving several models with strict latency objectives typically needs more. The cost scales with the number of distinct deployments, not with tokens, which is why consolidating models onto one fleet (for example through multi-LoRA serving) is an engineering saving as much as a GPU saving.

Redundancy and tail latency

An API absorbs hardware failures for you. Self-hosting means you hold spare capacity for them. Hu et al. (2024) document how often hardware and infrastructure failures interrupt jobs in a large LLM data centre, and Dean and Barroso (2013) explain why latency at the tail, not the median, drives the need for replicas: when a request fans out or retries, the slowest replica sets the user's experience. Spare replicas and multiple regions are the price of a latency and availability promise, and in this model they alone set the fleet size up to about 5 billion output tokens a month.

Compliance and data handling

Both options carry compliance cost, of different kinds. Self-hosting adds audit scope: access control on GPU hosts, logging that does not leak prompts, retention policies for request data and for the training sets behind your fine-tune. The API adds vendor review and often a redaction layer. In our ranges compliance moves break-even by about 15%. It decides the question only when a policy forbids sending data out, in which case the API row disappears and the relevant comparison becomes hosted-in-your-boundary versus on-prem.

Training, evaluation and model upgrades

The open options carry the cost of the fine-tune itself and of keeping it current: periodic retraining as data drifts, evaluation of each candidate, and a full re-run when a better base model appears. We amortised $4,000 a month. The ROI of post-training article treats these costs as an investment with a payback period.

On-premises hardware

Owned servers at the stated assumptions cost the equivalent of $1.88 per GPU-hour at full allocation, against $2.50 rented, but they come in eight-GPU units, need a two-server minimum for redundancy, and take more engineering. In this model on-prem beats rented dedicated GPUs only above 68 billion output tokens a month. Ownership makes sense when the fleet is large and steadily busy, for example when the same servers also run training (see the off-peak training section of the utilisation article), and when power and space are already paid for.

Preemptible capacity

Discounted preemptible GPUs lower $h$ at the cost of interruptions. SpotServe (Miao et al., 2023) reported 54% monetary savings against on-demand instances for LLM serving by adapting parallelism and migrating state when instances were reclaimed. That moves the GPU line, which is a minority of TCO below several billion tokens a month, and adds engineering complexity, so it pays most at the high-volume end of the chart. Griggs et al. (2024) make the complementary point that mixing GPU types for different request sizes can cut serving cost by up to 77% in conversational settings.

How TCO comparisons go wrong#

Comparing GPU cost with token cost. The most common error. A GPU-hour price times hours needed is a lower bound on one line item. Detect it by asking whether the self-host estimate contains a salary.

Sizing for average traffic. A fleet sized to average rate fails at peak. If the estimate's utilisation is above about 60% for interactive traffic, it has probably ignored peaks, headroom or failures.

Assuming the frontier API price is fixed. Per-token prices can change several times during the life of a hardware commitment, and the tornado shows that output price is the second-largest driver. Re-run the model with a lower price before committing to a multi-year hardware purchase.

Ignoring the hosted open-model option. The decision is rarely only "frontier API or our own GPUs". A per-token service for your fine-tuned open model removes the redundancy floor and most of the engineering, and in this model it is the cheapest option from 172 million output tokens a month up to beyond 100 billion.

Counting the fine-tune twice or not at all. Training cost belongs to the open options whether they are self-run or hosted. Leaving it out flatters both; putting it only on the self-run side flatters the hosted one.

Decision guide#

Implementation sketch#

import math
HOURS = 730

def fleet(V_out_m, R=2900, par=2.5, headroom=0.8, regions=2, spare=1):
    avg = V_out_m * 1e6 / (HOURS * 3600)                 # output tokens/s
    serve = math.ceil(avg * par / (R * headroom))
    g = regions * (math.ceil(serve / regions) + spare)
    return g, 1 - avg / (g * R)                          # GPUs, idle share

def tco_dedicated(V, h=2.50, fte=1.0, fte_cost=22000, compliance=3000, tools=1000, train=4000, **kw):
    g, idle = fleet(V, **kw)
    return HOURS * h * g + fte * fte_cost + compliance + tools + train

def tco_api(V, p_in, p_out, rho=3.0, fte=0.15, fte_cost=22000, compliance=1500, train=0):
    return V * (rho * p_in + p_out) + fte * fte_cost + compliance + train

def breakeven(f, g, lo=1, hi=1e6, n=4000):
    for i in range(n + 1):
        V = lo * (hi / lo) ** (i / n)
        if f(V) <= g(V):
            return V
    return math.inf

print(tco_dedicated(3000))                                        # 37,300
print(breakeven(tco_dedicated, lambda V: tco_api(V, 3.0, 15.0)))  # ~1,358 (million output tokens)

Put your own salary, compliance and traffic-shape numbers in before trusting any output. When POFE estimates serving cost for a trained model it reports the per-token figure and the fleet floor separately for this reason: they answer different questions.

Further reading#

Barroso, Hölzle and Ranganathan's The Datacenter as a Computer is the standard reference on data-centre cost modelling, including capital versus operating cost and the role of utilisation, and Barroso and Hölzle (2007) is the short paper on why servers spend most of their time well below peak load. Sculley et al. (2015) catalogue the maintenance costs of production ML systems. Dean and Barroso (2013) explain why tail latency forces redundancy. Hu et al. (2024) characterise failures and utilisation in an LLM data centre. Miao et al. (2023) and Griggs et al. (2024) show two ways to cut the GPU line: preemptible instances and heterogeneous GPU types. The break-even study has the simpler model this one extends, and the quality-adjusted comparison covers the case where the options are not equally accurate.

Frequently asked questions#

References#

1. Luiz André Barroso, Urs Hölzle, Parthasarathy Ranganathan (2019). The Datacenter as a Computer: Designing Warehouse-Scale Machines (3rd ed.). doi.org/10.1007/978-3-031-01761-2
2. Luiz André Barroso, Urs Hölzle (2007). The Case for Energy-Proportional Computing. doi.org/10.1109/MC.2007.443
3. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, et al. (2015). Hidden Technical Debt in Machine Learning Systems. papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
4. Jeffrey Dean, Luiz André Barroso (2013). The Tail at Scale. doi.org/10.1145/2408776.2408794
5. Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, et al. (2024). Characterization of Large Language Model Development in the Datacenter. arxiv.org/abs/2403.07648
6. Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, Zhihao Jia (2023). SpotServe: Serving Generative Large Language Models on Preemptible Instances. arxiv.org/abs/2311.15566
7. Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, Ion Stoica (2024). Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity. arxiv.org/abs/2404.14527

Related reading