Cost per task is what it costs, on average, to get one unit of work done to an acceptable standard: a ticket resolved, a document extracted, a pull request that passes review. For a single short model call it is close to the token bill. For anything agentic it is not. A task may take several model calls, each re-reading a growing context; it may call tools that cost money; it may fail and be retried; it may be escalated to a person; and it may fail silently and cost far more later. This article builds an expected-cost model for a task with all of these terms, derives it in closed form, runs it for a frontier-API agent, a post-trained open-model agent and a cascade of the two, and checks the averages with a Monte Carlo simulation. The headline result under our assumptions: model tokens were under 10% of the cost of a task, and the per-attempt success rate and the verifier's recall decided almost everything else.
Why per-token prices mislead for tasks#
A price per token answers how much a call costs. A business pays for outcomes. The two diverge in three ways. Tokens per task are not fixed: an agent that takes eight steps instead of four re-reads its context eight times. Attempts per task are not one: failures that a check catches are retried, and some are escalated. Outcomes are not all equal: an error that nobody catches is the most expensive line, and it is invisible in any token dashboard. Kapoor et al. (2024) made the evaluation version of this argument for AI agents: benchmark leaderboards that report accuracy without cost reward needlessly expensive agents, and cost should be optimised jointly with accuracy. Chen, Zaharia and Zou (2023) made the deployment version: once you price outcomes, routing and cascading between models of different cost becomes the main lever. This article supplies the arithmetic for both.
The model#
Tokens per attempt: context grows every step
An agent attempt makes model calls. The first call reads a fixed prefix of tokens (system prompt, tool schemas, task input). Each later call also re-reads everything accumulated so far, which grows by tokens per step (the previous output plus the tool result). Each call writes tokens. Total input tokens per attempt are
which is quadratic in the number of steps. With prefix caching, everything except the newest of each call is a cache hit, so only tokens are processed fresh and the remaining are billed at a fraction of the input price. With per-token prices and a per-call tool cost , one attempt costs
Attempts, escalation and silent failures
Each attempt succeeds with probability . A verifier (tests, a schema check, a rule engine or a judge model) catches a failed attempt with probability , its recall; an uncaught failure ends the task with a wrong result that costs later. A caught failure is retried, up to attempts in total, after which the task goes to a person at cost . The probability that an attempt ends in a caught failure is . The expected number of attempts and the outcome probabilities follow from a truncated geometric series:
The expected cost of a task is
For a cascade that runs a cheap model first and sends caught failures to a stronger model with success , the same algebra applies with the first attempt priced and scored as the cheap model and later attempts as the strong one. If you prefer cost per acceptable outcome, divide by , counting escalated tasks as resolved by the person.
When to hand a task to a person
Escalation is a decision with a known optimum. Chow (1970) showed that for a classifier with a reject option, the error-minimising rule is to reject when the highest posterior probability falls below a threshold set by the ratio of the cost of rejecting to the cost of an error. In this setting the rule reads: escalate when the model's calibrated probability of being right is below . With a person costing $6 per task and a silent error $15, escalate anything less than 60% likely to be right. Geifman and El-Yaniv (2017) developed the modern version for neural networks, selective classification, which trades coverage (the share of inputs the model answers) against risk (the error rate on those it answers) with a guaranteed bound. Both need calibrated confidence, which a post-trained model can be trained to produce and which most agents never measure.
Assumptions#
All values are illustrative assumptions for a support-resolution agent that reads a ticket, looks up account data through tools, and drafts an action. None is a measurement of a real product.
| Frontier API agent | Post-trained open agent | |
|---|---|---|
| Prefix , growth , steps , output per step | 4,000, 1,500, 8, 250 | 1,200, 1,200, 6, 150 (tool schemas and policy learned; fewer wasted steps) |
| Price per 1M input / output; cached-input fraction | $3 / $15; 0.10 | $0.10 / $0.40; 0.25 |
| Tool cost per call | $0.002 | $0.002 |
| Per-attempt success | 0.72 | 0.66 |
Shared: verifier recall , at most attempts, human handling (about six minutes of an agent's loaded time), silent failure (a wrong account action that must be found and reversed, plus customer cost). The cascade runs the post-trained agent first and the frontier agent on the retry.
Worked example by hand#
Frontier agent, one attempt of 8 steps. Input tokens: . Without caching that is ; add output and tools for . With caching, tokens are fresh () and are cached at 10% (), so the attempt costs : caching removed 60% of the model bill.
Outcomes: , so , , and . Expected cost:
The model and tools are 9% of the cost of a task. Silent failures are 70%.
The post-trained agent's attempt reads 25,200 input tokens over 6 steps, 7,200 of them fresh, and costs , of which is tool calls: at open-model prices the tools, not the tokens, are the main per-attempt cost. But with its expected task cost is : of model and tools, of escalation and of silent failures. Eight times cheaper per attempt, 20% more expensive per task. The cascade lands between them at .
Long contexts: the quadratic bill#
At 4 steps an attempt reads 25,000 input tokens; at 8, 74,000; at 12, 147,000; at 20, 365,000. Doubling the steps from 8 to 16 multiplies uncached input by 3.3. Three remedies follow from the formula. Prefix caching turns the quadratic term into a discounted one, which is why it is the single largest token saving for agents. Reducing (summarising tool results, returning only the fields the next step needs) shrinks the coefficient of the quadratic. Reducing (fewer wasted calls) attacks it directly, and is one of the things trajectory-level post-training buys: a model trained on successful tool-use traces takes fewer exploratory steps. Long contexts also cost accuracy: Liu et al. (2023) found that models use information in the middle of long inputs less reliably than information at the start or end, so a bloated context can lower as well as raise , and in this model is worth far more.
The variance of matters too. Running the attempt with a randomly distributed step count (negative binomial with mean 8, seed 5, 20,000 samples) gave a mean inference cost of $0.112 against $0.107 at exactly 8 steps, because cost is convex in . The 90th and 99th percentiles were $0.20 and $0.33: a few long-running tasks cost three times the median. Cap steps per attempt, and treat hitting the cap as a caught failure.
Success rate and verifier recall decide the cost#
Both agents' total cost falls steeply with , and the two curves nearly coincide, because inference is a small part of either. At equal success rates of 72%, the post-trained agent costs $1.35 per task against the frontier agent's $1.46; the whole difference is the token bill. At 80% the post-trained agent costs $0.87; at 90%, $0.38. The frontier agent at 90% costs $0.48. So the question "should we fine-tune to save on tokens" is usually the wrong one for agents. The right one is whether post-training raises on your tasks. Large Language Monkeys (Brown et al., 2024) showed the other way to raise task-level success: repeated sampling lifted the share of SWE-bench Lite issues solved by one open model from 15.9% with one sample to 56% with 250. That works when a verifier can pick the correct sample, and its cost is linear in samples, so it pays exactly when and are large relative to , which is the regime this model describes. Without a reliable verifier, they found majority voting and reward-model selection plateaued.
The sensitivity ranking is stark. The cost of a silent failure ($5 to $50) moves the cascade from $0.82 to $4.57 a task. Verifier recall (0.60 to 0.95) moves it from $2.62 to $0.89: a better check is worth more than any model change here, because it converts $15 silent failures into $0.11 retries or $6 escalations. The open model's success rate (0.55 to 0.80) moves it from $2.19 to $0.98, human handling cost from $1.41 to $2.51, and the frontier model's success from $1.91 to $1.38. The frontier output price across $5 to $25 per million, the number of steps from 3 to 12, and context growth from 500 to 4,000 tokens per step each moved the cascade by under 2%.
The distribution, not just the mean#
Expected cost hides a two-humped distribution. We simulated 20,000 tasks per configuration (seed 5) with random step counts and random outcomes. The median task cost $0.12 on the frontier agent, $0.016 on the post-trained agent and $0.018 on the cascade: most tasks succeed first time and cost only tokens and tools. The 90th percentile was about $6 in all three (a human escalation) and the 99th about $15 (a silent failure). The means were $1.49, $1.75 and $1.65, close to the closed-form values, and the share of tasks costing more than $1 was 12%, 16% and 14%. A dashboard showing median cost per task would report the post-trained agent as eight times cheaper than the frontier agent. The mean says it is 17% more expensive.
Measuring the inputs in production#
The model needs five numbers that no price list provides. Per-attempt success comes from a labelled sample of production tasks, scored by the same rubric your reviewers use, split by first attempt and retry because the two differ. Verifier recall needs the failures themselves: take attempts that a later human review marked wrong and count how many the verifier had flagged. Human handling cost is queue time multiplied by loaded hourly cost, which most support and operations teams already track. The silent-failure cost is the hardest and the most important; estimate it from incidents (time to detect, time to reverse, refunds or credits issued) and report it as a range rather than a point. Steps and tokens per attempt come from your traces. Refresh all five after every model or prompt change, since a change that lowers tokens by 20% and by two points is, by the sensitivity analysis above, a net loss.
How cost-per-task estimates go wrong#
Counting tokens only. The most common error, and the one this article is about. If your cost report has no line for escalations and no estimate for silent failures, it is a token report.
Assuming the verifier is perfect. With , silent failures vanish and the model reports much lower cost. Measure recall on a labelled sample of failures, not on synthetic ones.
Ignoring step-count variance. Cost is convex in and long-tailed. Report percentiles of steps per task and cap them.
Retrying without changing anything. Retrying the same model at the same temperature on the same context assumes failures are independent. Often they are not; the same input fails the same way. Estimate for the second attempt separately, or retry on a different model as the cascade does.
Using an uncalibrated confidence to escalate. Chow's threshold is only optimal if the probability is calibrated. Check calibration on held-out data before setting the threshold from cost ratios.
Which lever to pull#
| If the waterfall is dominated by | Pull this lever first |
|---|---|
| Silent failures | Raise verifier recall: tests, schema and policy checks, a trained judge; then escalate low-confidence outputs |
| Human escalation | Raise : post-train on successful trajectories, or cascade to a stronger model before the person |
| Retries | Change something between attempts: a stronger model, more context, a different tool plan |
| Model tokens | Prefix caching, smaller , fewer steps, then a cheaper or post-trained model |
| Tool calls | Batch or cache tool results; train the model to call fewer tools |
Implementation sketch#
def attempt_cost(K, s0, delta, out, p_in, p_out, kappa, tool):
total_in = K * s0 + delta * K * (K - 1) / 2
fresh = s0 + (K - 1) * delta
cached = total_in - fresh
return (fresh * p_in + cached * p_in * kappa + K * out * p_out) / 1e6 + K * tool
def task_cost(C_a, p, recall, R, H, c_silent):
d = (1 - p) * recall # caught failure per attempt
attempts = (1 - d ** R) / (1 - d)
return dict(
total=attempts * C_a + d ** R * H + (1 - p) * (1 - recall) * attempts * c_silent,
p_ok=p * attempts, p_human=d ** R, p_silent=(1 - p) * (1 - recall) * attempts)
Ca = attempt_cost(K=8, s0=4000, delta=1500, out=250, p_in=3.0, p_out=15.0, kappa=0.10, tool=0.002)
print(Ca, task_cost(Ca, p=0.72, recall=0.80, R=2, H=6.0, c_silent=15.0)) # 0.107, total about 1.46The inputs that matter most, and , come from evaluation, not from price lists. In POFE an evaluation run reports per-attempt success and the measured token and step counts for each candidate model, which is enough to fill in this function for every option you are considering.
Further reading#
Kapoor et al. (2024) is the argument for reporting agent cost alongside accuracy. Chen, Zaharia and Zou (2023) introduced LLM cascades for cost. Brown et al. (2024) quantify how repeated sampling with a verifier raises task success. Chow (1970) and Geifman and El-Yaniv (2017) give the theory of when to abstain and hand over. Liu et al. (2023) show why long contexts are not free in accuracy either. The per-token inputs come from cost per million tokens, the model comparison from the quality-adjusted cost model, and the decision to post-train for a higher from the ROI of post-training. For training agents on successful trajectories, see rejection-sampling fine-tuning and RL with verifiable rewards.
Frequently asked questions#
How do I calculate the cost per task of an LLM agent?
Expected cost = expected attempts × cost per attempt + probability of escalation × human cost + probability of an undetected failure × its cost. With caught-failure probability d = (1 − p) × recall, expected attempts over at most R tries are (1 − d^R) / (1 − d).
Why do agent token costs grow so fast with the number of steps?
Each step re-reads the accumulated context, so total input tokens are K × s0 + Δ × K(K − 1)/2, quadratic in the number of steps K. Prefix caching bills most of that at a discount; summarising tool results and taking fewer steps shrink it directly.
What share of agent cost is tokens?
In the illustrative support-agent model on this page, model tokens and tool calls were about 9% of expected cost per task for a frontier-API agent and about 1% for a post-trained open agent. Human escalations and undetected errors made up the rest.
When should an LLM task be escalated to a human?
By Chow's reject rule, escalate when the calibrated probability that the output is right falls below 1 − H / c_u, where H is the cost of human handling and c_u the cost of an undetected error. With H = $6 and c_u = $15, that threshold is 60%.
Is a cheaper model always cheaper per task?
No. In the model, a post-trained open agent eight times cheaper per attempt was 20% more expensive per task at a six-point lower success rate; at equal success it was cheaper, and at 80% success it cost $0.87 against $1.46.
References#
- Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan (2024). AI Agents That Matter. arxiv.org/abs/2407.01502
- Lingjiao Chen, Matei Zaharia, James Zou (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arxiv.org/abs/2305.05176
- Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arxiv.org/abs/2407.21787
- C. K. Chow (1970). On optimum recognition error and reject tradeoff. doi.org/10.1109/TIT.1970.1054406
- Yonatan Geifman, Ran El-Yaniv (2017). Selective Classification for Deep Neural Networks. arxiv.org/abs/1705.08500
- Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang (2023). Lost in the Middle: How Language Models Use Long Contexts. arxiv.org/abs/2307.03172