"Fine-tune a small model and save 90% on inference" is true at some volumes and false at others. The useful question is where the line is for your workload, and which of your assumptions it depends on. We built the simplest cost model that captures the real structure — per-token API pricing on one side, a capacity floor plus fixed costs on the other — and ran a sensitivity analysis on every input.
The model#
Let be monthly output tokens (in millions) and the input-to-output token ratio. A hosted API costs
A self-hosted fine-tuned model needs enough GPUs to serve the average output rate at a realistic utilisation , never fewer than a redundancy floor , plus a 15% allowance for prefill of input tokens:
where is the GPU hourly price, the one-off training project cost amortised over a year, and monthly operating overhead. Break-even is the where the two are equal.
| Assumption | Value | Why |
|---|---|---|
| API price, input / output | $2.50 / $10.00 per 1M | Representative frontier-class list pricing; substitute yours |
| Input:output ratio ρ | 3:1 | Typical for assistants with system prompts and context |
| GPU price h | $2.50/H100-hour | Mid-market on-demand H100 pricing |
| Throughput per GPU | 1,800 output tok/s | Batched serving of a tuned ~8B model; an assumption, not a benchmark |
| Utilisation u | 40% | Average vs peak; real traffic is bursty |
| Redundancy floor gmin | 2 GPUs | Survive one node failing |
| Training project T | $6,000 | Compute, labelling and evals for the first model |
| Operations O | $1,500/month | On-call, monitoring, retraining time |
Every number above is an assumption, chosen to be plausible and stated so you can replace it. Nothing on this page is a measurement of a real deployment. The script is in research/cost_models.py.
Result: break-even at about 427M output tokens a month#
Under these assumptions the self-hosted model breaks even at about 427M output tokens per month — roughly 1.7B total tokens including input. Below that, the API is cheaper, often by an order of magnitude: at 10M output tokens a month the API costs $175 against a self-hosted floor of $7,475. Above it, the gap opens fast: at 10B output tokens the API line reaches $175,000 a month versus $14,775.
The shape matters more than the exact crossing point. Self-hosting is a step function dominated by a fixed floor; the API is linear. That means the decision is mostly about whether your volume clears the floor, not about squeezing GPU prices.
Which assumptions move the answer#
We varied each input one at a time over a plausible range and recomputed the break-even volume.
Three findings were not what we expected going in:
- GPU throughput and utilisation had no effect on the break-even point in the tested range. At break-even volume the deployment is still sitting on its redundancy floor of 2 GPUs (3 after the prefill allowance), so a faster serving stack doesn't change cost until much higher volumes. Optimising tokens per second is a scale problem, not an adoption problem.
- The API's price and the input:output ratio dominate. Halving the API output price moves break-even from 427M to 598M. Long-context workloads (high ρ) favour self-hosting sooner because input tokens are cheap to serve yourself and not cheap to buy.
- Ops overhead matters more than training cost. A $1,500/month operations line moves break-even more than tripling the training budget does, because training is amortised and ops recur.
Past the floor: marginal cost per token#
Once traffic needs more GPUs than the redundancy floor, the self-hosted cost grows linearly with a slope set by GPU price, throughput and utilisation:
With our assumptions that is per million output tokens, plus the prefill allowance — against an API cost of per million output tokens once the 3:1 input is included. At scale the marginal cost ratio, not the floor, is what matters, and that ratio is where throughput and utilisation finally show up: doubling serving throughput halves .
That is also why optimisations like batching, FP8 weights and speculative decoding are worth doing late rather than early. Before the floor is exceeded they save nothing; after it they compound with volume.
Worked example: a support assistant#
A support assistant handling 40,000 conversations a day, averaging 6 model turns of 250 output tokens each, produces output tokens a month — about 4.2× the break-even volume. Under the stated assumptions the API line costs roughly $31,500 a month and the self-hosted model $7,475 (the nearest point on the curve). The same assistant at 2,000 conversations a day sits well below break-even, and the honest recommendation there is to stay on an API or use serverless fine-tuned inference.
What the model leaves out#
- Quality. The comparison assumes the tuned model is good enough to replace the API. If it is better on your task — common for narrow tasks after post-training — the value isn't only cost.
- Latency. A small dedicated model often has much lower latency than a shared API, which can be worth more than the cost difference.
- Serverless fine-tuned inference. Per-token pricing for hosted fine-tuned small models removes the floor entirely and moves break-even far lower. POFE's serverless tier exists for exactly this region of the curve.
- Multi-LoRA. Serving many fine-tuned adapters on one base model spreads the floor across use cases; see LoRA vs full fine-tuning.
To put your own numbers in, use the training cost calculator for the one-off side and the guide on fine-tuning cost for line items. For hardware choices, see H100 vs H200 vs B200.
Frequently asked questions#
Is fine-tuning cheaper than using an API?
At high, steady volume, usually yes; at low volume, usually no. In our model with representative prices, a self-hosted fine-tuned small model breaks even around a few hundred million output tokens per month, because self-hosting has a fixed capacity and operations floor.
What drives the cost of self-hosting a fine-tuned LLM?
Below a few billion tokens a month, the fixed floor dominates: minimum GPU count for redundancy and operations overhead. Throughput optimisations matter only once traffic needs more GPUs than the floor provides.
How do I reduce the break-even volume?
Use serverless per-token inference for fine-tuned models, share GPUs across many LoRA adapters, or consolidate several use cases onto one deployment so they share the fixed floor.
research/cost_models.py in our research repository (Python, numpy, fixed seeds). See the methodology notes.