Prompting is the cheapest experiment in machine learning: change some text, rerun the eval. Fine-tuning costs data, compute and process. The right default is to push prompting until it plateaus — but teams often stop too early, or keep paying for 4,000-token prompts long after a fine-tune would have been cheaper and better.
Find the prompt ceiling first#
Before fine-tuning, establish what the best prompt achieves on your eval. That number is the baseline any fine-tune has to beat, and the gap tells you whether training is worth it. A credible prompt baseline includes a clear instruction, 3–10 diverse examples chosen to cover your hard cases, an explicit output schema, and — for reasoning tasks — space to think before answering. If that baseline is within your quality bar, stop and ship it.
Signals that fine-tuning will pay off#
- The prompt is long and growing. Every rule you add to fix one failure costs tokens on every call and sometimes breaks another case.
- Format errors persist at a low but costly rate — 2% invalid JSON at a million calls a day is twenty thousand failures.
- Examples don't fit. You need hundreds of demonstrations to cover the space; a prompt holds ten.
- Cost or latency dominates. A fine-tuned small model with a short prompt is cheaper per call than a large model with a long one.
- Data can't leave your environment. Fine-tuning an open model you host keeps it in.
A worked token-cost comparison#
Suppose your prompt carries 3,000 tokens of instructions and examples, and the average request adds 500 tokens of input and produces 300 tokens of output. Per million requests, the prompt overhead alone is
At $2.50 per million input tokens that is $7,500 per million requests just to repeat the instructions. A fine-tuned model that has internalised them needs perhaps 200 tokens of system prompt, a 93% cut in that line, and can usually be a smaller model at a fraction of the per-token price. Run the full comparison with your own volume in our break-even model.
How much data does fine-tuning need?#
It depends on how far you are moving the model. These ranges are what we see across POFE projects for LoRA SFT on 8B–32B instruct models; treat them as starting points, and measure.
| Goal | Typical examples | Notes |
|---|---|---|
| Output format, schema, tone | 300–2,000 | Quality and coverage of edge cases matter more than count |
| Domain adaptation (vocabulary, conventions) | 2,000–20,000 | Include hard and ambiguous cases deliberately |
| New task skill (extraction, coding, SQL) | 5,000–50,000 | Often followed by RL with a verifier |
| Preference optimisation (DPO) | 5,000–50,000 pairs | On-policy pairs from your SFT model are worth more |
| RL with verifiable rewards | 1,000–20,000 prompts | Prompts must be at the edge of the model's ability |
The fastest way to know is a data-scaling curve: train on 10%, 25%, 50% and 100% of your data and plot the eval. If the curve is still rising at 100%, more data will help; if it is flat from 25%, spend on quality or a different method instead.
The middle path: prompt + small fine-tune#
The two aren't exclusive. A common production pattern keeps a short prompt for things that change (today's policy, the user's context) and fine-tunes the stable behaviour (format, tone, decision rules). The fine-tune makes the prompt shorter and more reliable; the prompt keeps the system easy to adjust without retraining.
Related: What is post-training?, fine-tuning vs RAG, LLM fine-tuning cost.
Frequently asked questions#
Is fine-tuning better than prompt engineering?
Not by default. Prompting is cheaper and faster to iterate. Fine-tuning wins when prompts become long and brittle, when format errors persist, when you need many examples, or when cost, latency or data residency matter.
How many examples do I need to fine-tune an LLM?
For format and tone, a few hundred to two thousand high-quality examples. For domain adaptation, a few thousand to twenty thousand. For new task skills, five to fifty thousand. A data-scaling curve on your own eval is the reliable way to know.
Can few-shot prompting match fine-tuning?
For simple tasks with a strong model, often yes. For tasks needing many examples, strict formats, or small fast models, fine-tuning usually wins.