Prompted open model vs fine-tuned open model vs frontier API: a quality-adjusted cost model

Teams choosing how to run a language-model feature usually compare three options: an open model used as is, steered only by a prompt; the same kind of open model after post-training on the task; and a commercial frontier API. Comparing them on price per token is the most common way to get the decision wrong, because the options differ far more in how often they succeed than in what a token costs. This article builds a quality-adjusted cost model: the expected cost of completing one task, counting inference, the cost of the failures each option produces, and the fixed costs each option carries, spread over monthly volume. It derives the break-even volumes in closed form, maps which option wins across volume and failure cost, runs a Monte Carlo over the uncertain inputs, and then covers the parts money does not capture: latency, control and data exposure.

The three options, precisely#

The prompted open model is an instruction-tuned open-weight checkpoint you run (or rent per token) without any training of your own. All task knowledge lives in the prompt: instructions, schema, and a handful of examples. It has no fixed project cost and the lowest per-token price, but long prompts and lower task accuracy.

The post-trained open model is the same class of checkpoint after supervised fine-tuning, often followed by preference optimisation or reinforcement learning with verifiable rewards, usually as a LoRA adapter. The instructions and examples move into the weights, so prompts shrink and accuracy on the task rises, at the price of a training project and ongoing maintenance. You hold the weights.

The frontier API is a large general model behind a per-token price. It needs no training and is usually the strongest model on unfamiliar or open-ended inputs, but it costs one to two orders of magnitude more per token, and your inputs leave your boundary.

A fourth option falls out of the analysis and turns out to matter most at scale: the post-trained model with a router that escalates a small share of hard inputs to the frontier API. This is the cascade idea of FrugalGPT (Chen, Zaharia and Zou, 2023), which reported matching the best single model's performance at up to 98% lower cost, and of RouteLLM (Ong et al., 2024), which reported cost reductions of over 2× without loss of response quality on standard benchmarks.

What the evidence says about quality#

The model's outputs depend most on the success rates you plug in, so it is worth knowing what published comparisons found. Bucher and Martini (2024) compared fine-tuned smaller models with large generative models used zero-shot across sentiment, stance, emotion and party-position classification, and found the fine-tuned models better in every case they tested. Hsieh et al. (2023) trained a 770M-parameter model with rationales distilled from a larger one and outperformed a few-shot prompted 540B-parameter model on a benchmark using 80% of the available data. Le Scao and Rush (2021) quantified how much a prompt is worth relative to training data and found it was often worth hundreds of labelled examples on classification tasks, which is the flip side: a prompt is a cheap substitute for a small dataset, not for a large one. The pattern across these results is consistent. On a narrow, well-specified task with a few thousand good examples, a post-trained small model tends to match or beat a much larger prompted one. On open-ended inputs far from the training distribution, it does not. Your own evaluation set, not these papers, should supply the numbers; see how many eval examples you need to measure a gap of a few points.

The quality-adjusted cost model#

Cost of one task

For option $i$, let $n_i^{\text{in}}$ and $n_i^{\text{out}}$ be the tokens per task, $p_i^{\text{in}}$ and $p_i^{\text{out}}$ the prices per token, $q_i$ the probability the output is acceptable, $c_f$ the expected cost of an unacceptable output, $F_i$ the fixed cost per month, and $N$ the tasks per month. The expected cost of one task is

The failure cost $c_f$ is whatever a bad output costs you: an agent's minutes to fix a misrouted ticket, a refund, a lost conversion, a re-run. It is the same for every option because it is a property of the task, not the model. If a verifier catches failures and you retry until success, the inference term becomes $k_i/q_i$ instead; the cost-per-task article treats retries and escalation in full. For the cascade with escalation share $e$, where the router's escalated set contains a fraction $\varphi$ of the post-trained model's failures and the frontier model solves a fraction $q_f^{\prime }$ of those hard cases,

Break-even volume in closed form

Setting $C_a(N)=C_b(N)$ for an option $a$ with higher fixed cost and lower variable cost gives

The denominator has two parts: the per-task inference saving and the per-task quality saving. When the success rates are equal, $c_f$ drops out and only token prices matter. When they differ, the quality term $(q_a-q_b)c_f$ is usually the larger one, because inference costs for these tasks are fractions of a cent while failures cost cents to dollars. If the denominator is negative, option $a$ never breaks even at any volume.

Assumptions for a support-triage task#

The base case is a ticket-triage task: read a customer message plus account context, return a category, priority and a short routed summary as JSON. Every value below is an illustrative assumption, not a measurement or a quote from any provider.

The failure cost is $1.50 per bad triage (about two minutes of an agent's time to re-route plus some customer delay), varied from $0.05 to $20 in the Monte Carlo. The cascade escalates $e=10\%$ of tasks, its router catches $\varphi =50\%$ of the post-trained model's failures in that 10%, and the frontier model solves $q_f^{\prime }=75\%$ of those hard cases, which gives $q_h=1-0.06\times (0.5+0.5\times 0.25)=0.9625$.

Worked example by hand#

Per-task inference costs: prompted $2500\times 0.10/10^6+250\times 0.40/10^6=\$0.00035$; post-trained $400\times 0.10/10^6+120\times 0.40/10^6=\$0.000088$; frontier $1200\times 3/10^6+200\times 15/10^6=\$0.0066$. The frontier call costs 75 times the post-trained one. Now failures at $1.50: prompted $0.20\times 1.50=\$0.30$, post-trained $0.06\times 1.50=\$0.09$, frontier $0.08\times 1.50=\$0.12$. Failures cost 18 to 1,000 times more than inference, depending on the option.

At $N=100,000$ tasks a month, fixed costs add $5500/10^5=\$0.055$ to the post-trained model and $1500/10^5=\$0.015$ to the frontier API. Totals: prompted $0.300, post-trained $0.145, frontier $0.142, cascade $0.127. The break-even between post-trained and frontier is

Notice where the denominator comes from: $0.0065 of token savings and $0.030 of quality savings. If the post-trained model were only as accurate as the frontier API, break-even would be $4000/0.006512=614,250$ tasks a month, more than five times higher. At a million tasks a month the totals are prompted $0.300, frontier $0.128, post-trained $0.096 and cascade $0.064, so the cascade costs half as much per completed task as the frontier API, and almost all of that saving is fewer failures, not cheaper tokens.

What the model shows#

Three regimes appear. Below about 9,000 tasks a month the prompted open model is cheapest because it has no fixed cost and its $0.30 failure bill is still smaller than amortised fixed costs. From there to about 80,000 the frontier API is cheapest: high accuracy, no training project. Above that, the post-trained model with escalation wins, and the post-trained model alone also beats the frontier API above 110,000. The post-trained model beats the prompted model at 26,000 tasks a month, because its 14-point accuracy gain is worth $0.21 a task.

The ordering depends on two numbers more than any others: volume and the cost of a failure. Sweeping both gives the map below.

Read the map along a diagonal. When failures cost a few cents or less, the prompted model wins up to one to several million tasks a month, after which the post-trained model wins on token price alone. When failures cost a dollar or more, the prompted model is never right except at tiny volume, the frontier API wins the middle, and the cascade wins at scale. The post-trained model on its own has a narrow band: high volume, cheap failures. In this parameterisation, adding escalation beats the plain post-trained model at high volume whenever a failure costs more than about three cents, and at $1.50 per failure it covers its extra fixed cost above about 45,000 tasks a month.

How much accuracy is a fine-tune worth?

With failures at $1.50, every extra point of accuracy over the frontier API pulls break-even down sharply: +2 points gives 110,000 tasks a month, +5 points gives 49,000. With failures at $10, the same +2 points gives 19,000 and +5 gives 7,900. A post-trained model that is two points worse than the frontier API never breaks even at $1.50 or $10 per failure, whatever the volume. With failures at $0.10 it still does, at about 890,000 tasks a month, because token savings carry it. The practical reading: measure the gap on your evaluation set before modelling anything else, and measure it with enough examples that the confidence interval is narrower than the gap.

Uncertainty: who wins across plausible worlds

We sampled 4,000 joint scenarios (seed 11) with every price, success rate and fixed cost uniform over the ranges in the table and the failure cost log-uniform from $0.05 to $20, and counted how often each option was cheapest at each volume.

At 10,000 tasks a month the prompted open model was cheapest in 54% of scenarios and the frontier API in 40%. At 100,000 the frontier API led with 43%, the cascade had 32% and the prompted model 22%. At a million, the cascade was cheapest in 71%, the frontier API in 20%. The plain post-trained model never exceeded 6% in any volume bucket, which says something specific: in this model, fine-tuning pays for itself mostly when you keep a stronger fallback, because the fallback removes the failures that dominate cost. The ROI article turns this into a payback period.

Latency, control and data exposure#

Three properties sit outside the dollar figure and often decide the question anyway.

Latency. Under the stated assumptions, end-to-end time for one triage was 6.5 s for the prompted model (a long prompt and a verbose answer), 3.1 s for the post-trained model and 3.6 s for the frontier API. The post-trained model wins not because it decodes faster per token but because it reads 2,100 fewer tokens and writes 130 fewer. Short outputs are a training target you can set; see constrained decoding for enforcing the JSON format at serve time. On dedicated hardware you also control the batch-latency trade (see cost per million tokens), which a shared API does not let you tune.

Control. With open weights you choose when the model changes. A frontier API's model can be updated or retired on the provider's schedule, and every update is an unplanned regression test for you. A post-trained model also lets you encode refusals, tone and output schema in the weights rather than in a prompt that a user message can override. The cost of control is that you own regressions too: base-model upgrades mean retraining, and nobody else is evaluating your adapter.

Data exposure. A frontier API receives your inputs on every call. That is acceptable for many workloads under a suitable contract and often unacceptable for regulated data, in which case the comparison is not a cost question. Self-hosting does not make data risk vanish. A fine-tuned model can memorise training examples: Carlini et al. (2021) extracted verbatim training sequences, including personal information, from a public language model by sampling and ranking its outputs, and Shokri et al. (2017) showed that membership in a training set can be inferred from a model's outputs. If you train on customer data, treat the weights as containing that data, deduplicate and scrub the training set, and restrict who can query the model.

How this analysis goes wrong#

Optimistic success rates

The model is linear in $q$, and $q$ usually comes from a small evaluation. A two-point error in $q$ at $1.50 per failure moves cost by $0.03 per task, five times the frontier API's entire inference cost. Use a held-out set drawn from production traffic, report confidence intervals, and re-measure after deployment.

Hidden fixed costs

A fine-tune is not a one-off. Base models are superseded, input distributions drift and labels change. Leaving maintenance out of $F_t$ moves break-even down by a factor equal to the ratio of total to project cost. The TCO article lists the line items, including the engineering time that most estimates omit.

Using one failure cost for all errors

A wrong priority on a ticket and a hallucinated refund policy do not cost the same. When errors are heterogeneous, split $c_f$ by error type and estimate the rate of each per option, because options differ in which errors they make, not just how many.

Treating the options as exclusive

The largest savings in this model come from combining options. If you evaluate only the three pure options you will usually pick the frontier API in the middle band and miss the cascade above it.

Decision guide#

Implementation sketch#

The model is a dozen lines. The point of writing it down is to force every number into the open where someone can disagree with it.

def task_cost(n_in, n_out, p_in, p_out, q, fixed, N, c_fail):
    """Expected $ per task: inference + failures + fixed/N. Prices per 1M tokens."""
    k = (n_in * p_in + n_out * p_out) / 1e6
    return k + (1 - q) * c_fail + fixed / N

def breakeven(a, b, c_fail):
    """Volume at which option a (higher fixed cost) matches option b."""
    k = lambda o: (o["n_in"] * o["p_in"] + o["n_out"] * o["p_out"]) / 1e6
    gain = (k(b) - k(a)) + (a["q"] - b["q"]) * c_fail
    return (a["fixed"] - b["fixed"]) / gain if gain > 0 else float("inf")

tuned    = dict(n_in=400,  n_out=120, p_in=0.10, p_out=0.40, q=0.94, fixed=5500)
frontier = dict(n_in=1200, n_out=200, p_in=3.00, p_out=15.0, q=0.92, fixed=1500)
print(breakeven(tuned, frontier, c_fail=1.50))   # about 109,553 tasks / month

Replace the success rates with measurements on one shared evaluation set, scored the same way for every option. In POFE an evaluation run produces exactly these inputs: success rate on held-out data plus the measured prompt and completion lengths of each candidate, which is what this formula needs.

Further reading#

Chen, Zaharia and Zou (2023) introduced the cascade framing and catalogued strategies for reducing API spend. Ong et al. (2024) trained routers from preference data and showed they transfer across model pairs. Bucher and Martini (2024) and Hsieh et al. (2023) are two careful comparisons of small trained models against large prompted ones, and Le Scao and Rush (2021) measured what a prompt is worth in data points. Carlini et al. (2021) and Shokri et al. (2017) are the starting points for what training on sensitive data exposes. For the per-token side of this model, see cost per million tokens; for the full cost of running the open options yourself, self-hosting TCO.

Frequently asked questions#

References#

1. Lingjiao Chen, Matei Zaharia, James Zou (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arxiv.org/abs/2305.05176
2. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, Ion Stoica (2024). RouteLLM: Learning to Route LLMs with Preference Data. arxiv.org/abs/2406.18665
3. Martin Juan José Bucher, Marco Martini (2024). Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification. arxiv.org/abs/2406.08660
4. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. arxiv.org/abs/2305.02301
5. Teven Le Scao, Alexander M. Rush (2021). How Many Data Points is a Prompt Worth?. arxiv.org/abs/2103.08493
6. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, Colin Raffel (2021). Extracting Training Data from Large Language Models. arxiv.org/abs/2012.07805
7. Reza Shokri, Marco Stronati, Congzheng Song, Vitaly Shmatikov (2017). Membership Inference Attacks against Machine Learning Models. arxiv.org/abs/1610.05820

Related reading