The question is usually framed as a choice. It isn't. Retrieval-augmented generation (RAG) puts relevant documents into the prompt at inference time; fine-tuning changes the model's weights. They fix different failures, and the most reliable production systems use both. The useful question is which failure you're actually seeing.
What each one changes#
RAG, as introduced by Lewis et al. (2020), combines a parametric model with a non-parametric memory: a retriever finds passages and the generator conditions on them. Knowledge lives outside the model and can be updated by re-indexing. Fine-tuning moves information and behaviour into the weights, where it is fast to use and hard to update.
| RAG | Fine-tuning / post-training | |
|---|---|---|
| Fixes | Missing or stale facts | Wrong format, tone, judgement, skill |
| Update cycle | Minutes (re-index) | Hours to days (retrain, evaluate, deploy) |
| Citations | Natural: show the retrieved passage | Not possible from weights alone |
| Per-request cost | Higher: long prompts with retrieved context | Lower: short prompts, smaller model possible |
| Latency | Retrieval + longer prefill | Shorter prompts, faster |
| Access control | Per-document, at query time | Coarse: everything in the weights is visible to everyone |
| Failure mode | Retrieves the wrong passage; ignores the right one | Confidently wrong when facts change; forgets general skills |
Diagnose the failure before choosing#
Pull 50–100 failing production examples and label each with one cause. The distribution tells you what to build.
- "It didn't know." The right fact wasn't available to the model. → RAG, or better retrieval.
- "It knew but didn't use it." The passage was in context and the model ignored it, contradicted it or mixed it with prior knowledge. → Fine-tune on examples of grounded answering (including "the documents don't say").
- "Wrong shape." Format, length, tone, schema violations, tool-call errors. → Fine-tune; SFT fixes this efficiently.
- "Wrong judgement." Plausible but incorrect decision under your policy (refund when it shouldn't, escalate too late). → Preference optimisation or RL with a policy reward.
- "Too slow or expensive." → Fine-tune and distil a smaller model; keep retrieval if facts change.
Using both: retrieval-aware fine-tuning#
The strongest pattern is to fine-tune the model on the RAG task itself: training examples contain the retrieved passages the retriever actually returns — including irrelevant and misleading ones — and target responses that cite the right passage or decline when the answer isn't there. This teaches the model to read context, resist distractors and abstain, which is exactly where base models are weakest.
{"messages": [
{"role": "system", "content": "Answer only from the documents. Cite [doc-id]. If they don't contain the answer, say so."},
{"role": "user", "content": "[kb-114] Refunds within 14 days of purchase...\n[kb-208] Annual plans renew...\n\nCan I get a refund after 20 days?"},
{"role": "assistant", "content": "No — refunds are available within 14 days of purchase [kb-114]. After that, you can cancel renewal [kb-208]."}
]}Include 10–20% examples where the retrieved documents do not contain the answer and the correct response is to say so. Without them, the model learns that context always contains an answer and fabricates one.
Cost and latency#
RAG's costs are per request: retrieved context makes prompts longer, often 2–10× the user's question, and you pay for those input tokens on every call. Fine-tuning's costs are up front and fixed. At high volume, a fine-tuned small model with a short, focused retrieval step usually beats a large general model with a long context on both cost and latency. Our break-even model shows the input:output token ratio is one of the strongest drivers of when self-hosting pays off — and RAG raises that ratio.
Governance and data access#
If different users may see different documents, keep that knowledge in retrieval, where access control is enforced per query. Anything trained into weights is potentially extractable by anyone who can query the model. For regulated data, the safe pattern is: fine-tune on de-identified or synthetic examples of the behaviour, and retrieve the sensitive content under the user's permissions at inference time.
Decision table#
| Your situation | Start with |
|---|---|
| Facts change weekly or faster | RAG |
| Answers must cite sources | RAG (+ fine-tune for grounded answering) |
| Output format or tone is wrong | Fine-tuning (SFT) |
| Model has the context but answers wrongly | Fine-tune on the RAG task |
| Per-user document permissions | RAG |
| Need lower cost or latency at volume | Fine-tune + distil, keep light retrieval |
| Specialised reasoning (coding, SQL, extraction) | Fine-tuning, possibly RL with verifiers |
More: What is post-training?, fine-tuning vs prompt engineering, and post-training for customer support.
Frequently asked questions#
Should I use fine-tuning or RAG?
Use RAG when the model lacks facts, especially facts that change or need citations. Use fine-tuning when the model has the information but behaves wrongly: format, tone, judgement or skill. Many production systems use both.
Can fine-tuning replace RAG?
Not for changing or permission-controlled knowledge. Fine-tuning is slow to update and can't enforce per-user access. It can make a model far better at using retrieved context.
Is RAG cheaper than fine-tuning?
RAG has low up-front cost but higher per-request cost because of long prompts. Fine-tuning has an up-front cost but can reduce per-request cost, especially with a smaller distilled model. At high volume fine-tuning usually wins; at low volume RAG usually does.
References#
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. arxiv.org/abs/2005.11401