What is post-training? A complete guide for LLM teams

Pre-training gives a language model knowledge and fluency by predicting the next token across trillions of tokens of text. Everything that turns that raw predictor into something useful — following instructions, refusing what it should refuse, writing your JSON schema, solving the task you care about better than a model ten times its size — is post-training. This guide explains each stage of a modern post-training pipeline, what it changes in the model, what data and compute it needs, and how to tell whether it worked.

Post-training vs fine-tuning vs pre-training#

The terms overlap, so it helps to be precise.

Fine-tuning is the general act of continuing to train a pre-trained model on new data. Post-training is fine-tuning with a purpose and a pipeline: the sequence of stages labs use to turn a base model into an assistant, and that companies now use to turn an open assistant into a specialist. When people say "we fine-tuned Llama", they usually mean one post-training stage, almost always supervised fine-tuning.

The compute asymmetry is the reason post-training is accessible. Using the standard estimate of about $6ND$ FLOPs to train $N$ parameters on $D$ tokens (Kaplan et al., 2020), supervised fine-tuning of an 8B model on 50 million tokens costs

which is under two H100-hours at realistic utilisation. Pre-training the same model costs millions of times more.

The stages of a modern pipeline#

Instruction-tuned models from major labs are produced by some variant of this sequence; InstructGPT (Ouyang et al., 2022) made the first three stages standard, and reasoning models added RL with verifiable rewards.

1. Supervised fine-tuning (SFT) on demonstrations of the target behaviour.
2. Preference optimisation — RLHF with a reward model and PPO, or direct methods such as DPO — on comparisons between responses.
3. Reinforcement learning with verifiable rewards (RLVR) — GRPO and relatives — on tasks where correctness can be checked automatically.
4. Distillation of a large tuned model into a smaller one for cost and latency.
5. Evaluation and release gating at every step.

Teams adapting an open instruct model rarely need every stage. The decision guide in SFT vs DPO vs GRPO covers which to use when; the sections below explain what each one actually does.

Stage 1: supervised fine-tuning#

SFT minimises the negative log-likelihood of target responses $y$ given prompts $x$, computing loss only on the response tokens:

It is the same objective as pre-training, applied to a narrow, high-quality distribution. Its strength is teaching shape: output formats, tool-call schemas, domain vocabulary, tone. Its limitation is that it only ever sees good examples, so it learns to imitate their surface without learning what separates a good answer from a nearly-good one.

Three practical facts determine most SFT outcomes:

• Data quality dominates quantity. A few thousand expert-approved examples usually beat a hundred thousand scraped ones. De-duplication alone often removes 10–30% of support or email data.
• Narrow SFT causes forgetting. Training on a single task degrades general ability; mixing 5–20% general instruction data, or using adapters, limits it. Biderman et al. (2024) find LoRA "learns less and forgets less" than full fine-tuning — a trade-off, not a free lunch. See catastrophic forgetting.
• Chat templates must match serving. A model trained with one template and served with another fails in ways offline evals don't show.

Stage 2: preference optimisation#

Preference methods learn from comparisons: for a prompt $x$, a chosen response $y_w$ and a rejected one $y_l$. Classic RLHF fits a reward model $r_{\varphi }$ to these comparisons under the Bradley–Terry model and then optimises the policy with PPO against

Direct Preference Optimization (Rafailov et al., 2023) solves the same objective in closed form and trains on pairs directly, without a separate reward model or sampling. It is simpler, cheaper and now the default first choice. Our DPO research note derives the loss and shows in simulation what β controls and why chosen-response likelihoods fall during training.

Preference data teaches judgement: when to refuse, how long to be, which of two plausible answers is actually correct under your policy. The best pairs are on-policy — generated by the current model and labelled by people or a calibrated judge — and the worst are near-duplicates, which carry little signal and drive likelihood displacement.

Stage 3: reinforcement learning with verifiable rewards#

When correctness can be checked by code — a unit test, a SQL result set, a schema validator, a math answer — the model can learn from its own attempts. GRPO (Shao et al., 2024) samples a group of completions per prompt, scores them, and pushes probability toward completions that beat the group mean, without PPO's value network.

RLVR is where post-training produces capabilities SFT can't: multi-step reasoning, tool use, agents that recover from errors. It is also where it fails most expensively. Models optimise exactly the reward they are given, so a checker with a loophole produces a model that exploits it — reward hacking. And most of the compute goes into generating rollouts, which makes RL 3–6× more expensive per example than SFT.

A specific trap: prompts the model always solves or never solves produce groups with identical rewards and zero gradient. Our GRPO group-size simulation quantifies how much of a rollout budget that wastes and how group size trades against prompt coverage.

Stage 4: distillation#

Distillation trains a small student on outputs of a large teacher for your task. In its simplest form it is SFT on teacher completions; on-policy variants let the student generate and have the teacher score or correct it. For most companies this is where the economics of post-training come from: a tuned 8B student that matches a frontier model on one task can cost an order of magnitude less per token and respond several times faster. Whether that pays for itself depends on volume; our break-even model works through it.

Evaluation runs through every stage#

Every stage above can make a model look better while making it worse, and only evaluation catches it. A post-training eval suite needs:

• A frozen golden set drawn from production-like inputs, large enough to detect the gains you care about. Detecting a 3-point improvement reliably takes thousands of items, not hundreds — see how many eval examples you need.
• Judges calibrated against human labels, with sensitivity and specificity reported, not only κ — see what κ hides.
• Contamination checks between training and eval data.
• Slice-level reporting with confidence intervals, and release gates that block regressions.

The release checklist turns this into a list you can copy.

Adapters, memory and hardware#

Most post-training today uses low-rank adapters (LoRA): the base weights are frozen and small rank-$r$ matrices are trained in their place, so a weight update $\Delta W\in {\mathbb{R}}^{d\times k}$ is parameterised as $BA$ with $B\in {\mathbb{R}}^{d\times r},A\in {\mathbb{R}}^{r\times k}$ (Hu et al., 2021). Memory falls from about 16 bytes per parameter to about 2, and many adapters can be served on one copy of the base model. How much GPU memory fine-tuning needs has the full accounting; LoRA vs full fine-tuning covers quality.

When post-training is the wrong tool#

Post-training changes behaviour; it is a poor way to add facts that change. If the model needs to know this week's prices or a customer's account history, retrieval is the right tool, and fine-tuning is at best a complement — see fine-tuning vs RAG. If a well-written prompt with a few examples already reaches your quality bar at acceptable cost, that is the cheaper path — see fine-tuning vs prompt engineering. Post-training pays off when you need consistent behaviour at volume, lower cost or latency, data that can't be sent to a third-party API, or capability the prompt can't elicit.

Rule of thumb: prompt first, retrieve for facts, post-train for behaviour, distil for cost.

Getting started#

1. Write the eval before anything else, and score your current system on it.
2. Collect 1,000–5,000 high-quality demonstrations and run LoRA SFT on an 8B–32B open instruct model.
3. Generate pairs from that model, label them, and run DPO.
4. If correctness is checkable, add GRPO with a verifier.
5. Distil, deploy behind a canary, and feed production failures back into the eval.

The POFE workspace demo walks through steps 2–5 in your browser in about three minutes, and the glossary defines every term used here.

Frequently asked questions#

References#

1. Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361. arxiv.org/abs/2001.08361
2. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155. arxiv.org/abs/2203.02155
3. Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. arxiv.org/abs/2305.18290
4. Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. arxiv.org/abs/2402.03300
5. Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. arxiv.org/abs/2106.09685
6. Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. arXiv:2405.09673. arxiv.org/abs/2405.09673

Related reading