Every post-training project starts with the same question: which method? The honest answer is that SFT, DPO and GRPO are not competitors so much as stages, each good at a different kind of improvement and each with a different data requirement and compute bill. This guide explains what each method actually optimises, what data it needs, what it costs, and how to decide — with a decision table you can use on your own project.
The short answer
Use SFT when you can write down what a good answer looks like and the model mostly lacks format, domain vocabulary or style. Use DPO (or a sibling like ORPO, KTO or SimPO) when the model can already produce good answers but doesn't reliably prefer them over plausible bad ones, and you have — or can generate — comparisons. Use GRPO or another online RL method when correctness can be checked automatically (code runs, SQL returns the right rows, a math answer matches, a tool call succeeds) or when you have a reward model you trust, and the task rewards multi-step reasoning.
In practice, the strongest production models use all three in sequence: SFT to teach the shape of the task, preference optimization to sharpen judgement, and RL to push past what demonstrations alone can teach.
| Method | Learns from | Best at | Relative compute |
|---|---|---|---|
| SFT | Demonstrations (prompt → ideal response) | Format, style, domain language, tool-call schema | 1× |
| DPO | Pairs (chosen vs rejected) | Tone, safety boundaries, borderline judgement | ~2× |
| KTO | Unpaired good / bad labels | Learning from thumbs-up/down product feedback | ~1.5× |
| GRPO | Rewards on sampled outputs | Reasoning, code, SQL, agents, verifiable tasks | 3–6× |
Supervised fine-tuning (SFT)
SFT is next-token prediction on curated examples. You show the model a prompt and the response you want, compute cross-entropy loss on the response tokens only, and update the weights. It is the same objective as pre-training, applied to a small, high-quality, task-specific corpus.
What it is good at
SFT is unmatched for teaching the shape of a task: JSON schemas, citation formats, the voice of your support team, the vocabulary of ICD-10 or maritime law, when to call which tool. It is also the cheapest and most stable method, and the right first step in almost every project because every later method works better from a good SFT checkpoint.
Where it falls short
SFT can only teach what is in the demonstrations, and it treats every token in a demonstration as equally correct. It does not learn from mistakes, because it never sees them. That is why SFT models often look right but plateau: they imitate the surface of expert answers without learning the judgement that separates a good answer from a nearly-good one. SFT on too narrow a dataset also causes forgetting — the model gets better at your task and worse at general instruction following.
Data requirements
Quality dominates quantity. For a narrow task, 1,000–5,000 excellent examples often beat 100,000 mediocre ones. Remove near-duplicates, filter for responses that experts would sign off, and keep a slice of general instruction data (5–20%) to limit forgetting.
Direct preference optimization (DPO)
DPO trains the model to raise the probability of a chosen response relative to a rejected one for the same prompt, while staying close to a reference model (usually your SFT checkpoint). Mathematically it optimises the same objective as classic RLHF with a KL-regularised reward, but without training a separate reward model or running online sampling. That makes it much simpler and cheaper than PPO-style RLHF.
What it is good at
DPO teaches discrimination. Where SFT says "write like this", DPO says "this one, not that one". It is effective for tone, helpfulness versus verbosity, safety boundaries (refuse this, don't refuse that), and the borderline cases where two responses both look reasonable but one is subtly wrong — the refund that shouldn't be offered, the clause that is extracted but misattributed.
Where it falls short
DPO is offline: it learns only from the pairs you give it, not from the model's own current mistakes. If your pairs come from a different model, the gains can be smaller than expected. DPO can also over-optimise — pushing down the likelihood of both responses, producing longer outputs, or drifting in style. Watch length and the log-probabilities of chosen responses, not just the loss.
Variants worth knowing
- ORPO combines SFT and preference loss in one stage, with no reference model. Good when you want one run instead of two.
- KTO works with unpaired binary feedback — perfect for thumbs-up/down logs.
- SimPO uses length-normalised rewards and no reference model; often more stable on length.
- IPO adds regularisation that reduces overfitting to deterministic preferences.
Data requirements
Typically 5,000–50,000 pairs. The best pairs are on-policy: generate two or more responses from your current SFT model, then label which is better with human reviewers or a calibrated judge. Pairs where both responses are plausible teach far more than pairs where the rejected one is obviously bad.
GRPO and reinforcement learning
Group Relative Policy Optimization samples several responses for each prompt, scores each with a reward function, and updates the model to favour responses that scored above the group average. Unlike PPO it needs no separate value network, which roughly halves memory and makes RL practical on a single node for mid-sized models. It became the default RL method for reasoning after open reasoning models showed that RL with verifiable rewards produces substantial gains on math and code.
What it is good at
GRPO learns from the model's own attempts, including its mistakes, against an objective you define. That makes it the right tool when correctness is checkable: does the SQL return the expected rows, does the code pass the tests, does the extracted JSON validate and match the ground truth, did the agent complete the task in the sandbox. It is also how you teach multi-step reasoning, because the model is rewarded for outcomes rather than for imitating a particular chain of thought.
Where it falls short
RL optimises exactly what you reward, which is both its power and its danger. Reward functions get hacked: models learn to produce outputs that satisfy the checker without solving the task, or that please an LLM judge with confident phrasing. GRPO is also the most expensive method — most of the compute goes into generating rollouts, not gradient updates — and the most sensitive to hyperparameters such as group size, KL coefficient and learning rate.
Making it work
- Start from a good SFT checkpoint. RL amplifies what the model can already sometimes do; it rarely invents capability from zero.
- Prefer verifiable rewards over learned ones. Where you must use an LLM judge, calibrate it against human labels first.
- Combine rewards: correctness plus format plus a penalty for known failure modes (for example, offering refunds outside policy).
- Read the rollouts while training. Reward going up with eval scores flat is the signature of reward hacking.
- Pick prompts of the right difficulty. If the model solves 0% or 100% of a group, there is no learning signal.
How to decide: a practical framework
Answer four questions in order.
- Can you write ideal answers? If yes and the base model gets the format or domain wrong, start with SFT. Almost every project should.
- Is the remaining error about judgement between plausible answers? If reviewers can easily say which of two answers is better but struggle to write the perfect one, add DPO or KTO.
- Can correctness be checked by code? If yes, GRPO with a verifier is likely to give the largest gains, especially on reasoning-heavy tasks.
- Is cost or latency the main problem? Distil a strong teacher into a small model with SFT first; the rest is optimisation.
| Use case | Recommended path |
|---|---|
| Customer support agent | SFT on resolved tickets → DPO on reviewer preferences → GRPO with policy-compliance reward |
| Document extraction to JSON | SFT on labelled documents; GRPO with a schema + field-match reward if accuracy plateaus |
| Text-to-SQL / code | Short SFT for dialect and schema → GRPO with execution verifier |
| Brand voice / writing | SFT on approved content → DPO on editor choices |
| Medical / legal coding | SFT on expert-coded records → DPO from coder corrections |
| Replacing an expensive API | Distil with SFT on teacher outputs, then targeted DPO on failures |
Whatever you pick, evaluate first
The method matters less than the eval. Before training anything, build a held-out test set that reflects production traffic, define the metric you will ship on, and run the base model and your current solution through it. Without a baseline you can't tell whether a method helped, and without slice-level results you won't notice that GRPO improved the average while breaking non-English requests.
We cover this in depth in Evaluating fine-tuned models. For cost planning across methods, see How much does it cost to fine-tune an LLM?, and for adapter choices see LoRA vs full fine-tuning.
Pick the cheapest method that moves your eval. Then pick the next one.
You can try all three methods side by side on sample data in the POFE interactive demo — switch the method and watch the eval change.
Frequently asked questions
Is DPO better than SFT?
Not better — different. SFT teaches what a good answer looks like; DPO teaches the model to prefer good answers over plausible bad ones. DPO usually works best on top of an SFT checkpoint, not instead of one.
Is GRPO better than PPO?
For most verifiable-reward tasks GRPO matches PPO's results with less memory and simpler tuning because it needs no value model. PPO can still be preferable with a learned reward model on open-ended tasks, or when you need fine-grained per-token credit assignment.
How much data do I need for DPO?
Typically 5,000 to 50,000 preference pairs. On-policy pairs generated from your own SFT model and labelled by humans or a calibrated judge are worth far more than off-policy pairs from another model.
Can I skip SFT and go straight to RL?
Sometimes, with strong instruction-tuned base models and verifiable rewards. But a short SFT stage almost always makes RL faster and more stable, because the model already produces the right format and gets a non-zero reward more often.