Supervised fine-tuning (SFT), explained

Supervised fine-tuning (SFT) takes a pretrained language model and continues training it with the same next-token cross-entropy loss used in pretraining, but on a curated set of (prompt, response) demonstrations, and usually with the loss counted only on the response tokens. It is the first stage of almost every post-training pipeline: it turns a document completer into something that follows instructions, uses a chat template, stops when it should, and produces the format downstream stages expect. Preference optimisation (DPO and relatives) and reinforcement learning (PPO, GRPO, RLVR) both start from an SFT checkpoint and both use it as the reference policy their KL penalty points at. Get SFT wrong and everything after it inherits the problem.

Where SFT came from and what problem it solves#

A pretrained model has learned $p(\text{next token}\mid \text{prefix})$ over web text. Asked "Write a haiku about rain", it may continue with a list of other writing prompts, because that is what such a line is often followed by on the internet. The knowledge needed to write the haiku is there; the behaviour of answering is not. SFT supplies the behaviour by showing the model many examples of the conversation shape you want.

The idea of fine-tuning a pretrained network on labelled data is as old as transfer learning, but its modern form for LLMs crystallised in three steps. Wei et al. (2021) showed with FLAN that fine-tuning a 137B model on more than 60 NLP tasks phrased as natural-language instructions improved zero-shot performance on held-out task types. Chung et al. (2022) scaled this to 1,836 tasks and showed gains held across model families and sizes. Ouyang et al. (2022) made SFT the first of the three InstructGPT stages: about 13k prompts with responses written by labellers, followed by a reward model and PPO. The open community then showed how far data quality could carry the method: Self-Instruct (Wang et al., 2022) bootstrapped instruction data from the model itself, LIMA (Zhou et al., 2023) got a strong 65B chat model from just 1,000 carefully selected examples, and Tülu 3 (Lambert et al., 2024) published a 939,344-example SFT mixture spread over general chat, knowledge, maths, code, safety, multilingual and precise instruction following.

LIMA's "superficial alignment hypothesis" frames what SFT does: most capability comes from pretraining, and SFT mostly teaches which subdistribution of formats and styles to use. That is too strong as a universal claim (SFT on large reasoning traces clearly teaches new skills, as distilled reasoning models show), but it is the right default intuition for chat-style SFT. It explains why a thousand excellent examples can beat a hundred thousand mediocre ones.

The objective#

Let $\mathcal{D}=\{(x^{(i)},y^{(i)}){\}}_{i=1}^N$ be demonstrations, where $x$ is the prompt (system message, user turns, any earlier assistant turns, all rendered through the chat template) and $y=(y_1,\dots ,y_T)$ is the target response of $T$ tokens. The model ${\pi }_{\theta }$ is autoregressive. SFT minimises the negative log-likelihood of the response given the prompt:

Here $y_{<t}$ is the ground-truth prefix of the response, not the model's own sample: training is teacher-forced, so every position is computed in parallel in one forward pass. In code, the sum is computed over all positions of the concatenated sequence $[x;y]$, with a mask $m_t\in \{0,1\}$ that is 1 on response tokens and 0 on prompt and padding tokens:

where $s^{(i)}$ is the full token sequence of example $i$. The normaliser $Z$ matters more than it looks; we come back to it below.

Why minimising NLL is the right thing (and what it implies)

Minimising expected NLL over $\mathcal{D}$ is the same as minimising the forward KL divergence from the data distribution to the model, because the data entropy term does not depend on $\theta$:

Forward KL is mass-covering: the model is punished heavily for assigning low probability to anything that appears in the data, and only mildly for putting probability on things that never appear. Two practical consequences follow. First, every bad example in your data is something the model is forced to learn to produce; there is no mechanism to down-weight it. Second, SFT never sees the model's own mistakes, because it conditions on gold prefixes. At inference the model conditions on its own outputs, and an early slip can push it into states no demonstration covered. This train/test mismatch (exposure bias) is exactly what on-policy methods address: on-policy distillation, rejection-sampling fine-tuning and RL all train on samples from the current policy.

The gradient is simple and worth seeing. For logits $z_t\in {\mathbb{R}}^V$ at position $t$ and softmax output $p_t$, the gradient of the token loss with respect to the logits is

where $e_{y_t}$ is the one-hot vector of the target. Every token pushes its own logit up by $1-p_t[y_t]$ and every other logit down in proportion to its probability. Tokens the model already predicts confidently contribute almost nothing; surprising tokens dominate the update.

SFT next to its siblings

Many post-training objectives are SFT with a weight in front of the log-likelihood. Writing them side by side makes the family resemblance clear:

So SFT is the special case where every example gets weight one and the data never changes. Rejection sampling (RFT) makes the weight a 0/1 filter; REINFORCE makes it a signed advantage; distillation replaces the one-hot target with a full teacher distribution. Preference methods such as DPO are different in kind: they contrast two responses rather than imitating one.

Worked example: the loss on one short response#

Take the prompt "What is 2+2?" and the target response "4" followed by an end-of-turn token. With a chat template the sequence is something like <|user|> What is 2+2? <|assistant|> 4 <|end|>. Suppose the response has three supervised tokens (say a leading space token, "4" and the end token) and the model assigns them probabilities 0.5, 0.25 and 0.8 at their positions. The token NLLs are

for a sum of 2.303 nats and a per-token mean of 0.768 nats (we computed these in numpy; perplexity on this response is $e^{0.768}\approx 2.15$). The prompt tokens contribute nothing because their mask is zero. The gradient at the "4" position pushes that logit up by $1-0.25=0.75$, three times harder than the end token at $1-0.8=0.2$. This is the whole of the SFT update; everything else is bookkeeping about which tokens are masked and how losses are averaged.

Loss normalisation: a silent hyperparameter

Batches mix short and long responses, and how you average decides how much each one counts. Take a batch with a 4-token response whose tokens each have loss 2.0 and a 400-token response whose tokens each have loss 1.0. Averaging over all supervised tokens in the batch gives a loss of 1.010, and the short example carries 0.99% of the weight. Averaging per sequence first and then across sequences gives 1.5, with the short example carrying 50%. Neither is wrong, but they train different models: token-level averaging favours long responses (good when long reasoning traces are the point), sequence-level averaging gives every conversation an equal vote (better when many short, crisp answers matter). Gradient accumulation adds a third trap: if each micro-batch is averaged over its own token count and the micro-batches are then averaged, the effective weighting depends on how examples happen to fall into micro-batches. Normalise by the total token count across the whole accumulated batch if you want token-level averaging to mean what it says.

The algorithm, step by step#

for each example (messages, ...) in dataset:
    ids = chat_template(messages)                      # system, user, assistant turns
    labels = ids.copy()
    labels[not assistant_token] = IGNORE (-100)        # mask prompt, earlier turns if desired, padding
pack or pad into batches of ~max_seq_len tokens

for step in range(total_steps):
    batch = next(loader)
    logits = model(batch.ids, attention_mask=batch.mask)     # one teacher-forced forward pass
    loss = cross_entropy(logits[:, :-1], batch.labels[:, 1:], ignore_index=-100,
                         reduction="sum") / n_supervised_tokens_in_global_batch
    loss.backward()
    clip_grad_norm(model, 1.0)
    optimizer.step(); scheduler.step(); optimizer.zero_grad()
    every k steps: eval held-out NLL, run generation evals, save checkpoint

Three details distinguish a good SFT run from a merely working one. Templates: render training data with exactly the chat template you will serve with, including the end-of-turn token, and supervise that end token, or the model will not learn to stop. Multi-turn masking: in a conversation with several assistant turns you can supervise all of them (common) or only the last; supervising earlier turns written by a weaker system will teach their flaws. Packing: concatenating short examples into full-length sequences removes padding waste, but without per-document attention masks the model attends across example boundaries, which is mostly harmless for short chat data and harmful for long-context work. Most modern trainers support packing with position-id resets or block-diagonal attention.

The data SFT needs

The format is a list of messages per example, with roles (system, user, assistant, optionally tool) and a single target assistant turn or several. Sizes that work in practice span three orders of magnitude: a few hundred to a few thousand examples to teach a format, tone or narrow task on a strong base; 10k to 100k for a domain assistant; several hundred thousand to millions for a general-purpose chat model built from a base model (Tülu 3's 939k is a published reference point). Quality and diversity dominate quantity: LIMA's 1,000 examples were chosen for stylistic consistency and prompt diversity. Deduplicate near-duplicates, decontaminate against your evaluation sets, and check that the distribution of prompt types matches what the model will see in use. If you are teaching a model to reason, the responses must contain the reasoning; SFT can only imitate what is on the page.

Hyperparameters that matter#

Two of these deserve comment. The number of epochs interacts with held-out loss in a counter-intuitive way. Ouyang et al. (2022) report that their SFT models overfit on validation loss after one epoch, yet training for 16 epochs improved reward-model score and human preference ratings. Validation NLL measures how well the model covers the reference responses; a model that becomes more decisive (lower entropy) can be both worse at NLL and better to read. Judge SFT checkpoints on sampled outputs. The learning rate is the main lever on forgetting: higher rates move further from the base model, which helps a narrow task and hurts everything else.

Prompt-loss weighting is a smaller knob with mixed evidence. Huerta-Enochian and Ko (2024) found that, when training on short-completion data, a small prompt-loss weight (0.01 to 0.5) gave better multiple-choice and short-generation results than full masking, while a weight near 1.0 did better on long generation; and Shi et al. (2024) found that applying the loss to instructions as well (their "instruction modelling") helps most when instructions are long relative to outputs or the dataset is small, attributing the gain to reduced overfitting. With typical chat data and responses longer than prompts, masking the prompt remains the standard default. NEFTune (Jain et al., 2023), which adds uniform noise to embeddings during training, raised Llama-2-7B fine-tuned on Alpaca from 29.79% to 64.69% on AlpacaEval in its paper; it is a cheap thing to try when your outputs feel terse.

Compute and memory cost#

SFT is the cheapest stage of post-training per example: one forward and one backward pass, no sampling, no second model. Compute is roughly $6PD$ FLOPs for $P$ parameters and $D$ training tokens (counting all tokens processed, including masked prompt tokens, since they still go through the network). A 100M-token SFT run on an 8B model is about $6\times 8\times 10^9\times 10^8\approx 4.8\times 10^{18}$ FLOPs, which is a few hours on a single 8-GPU node at realistic utilisation.

Memory for full fine-tuning with AdamW in mixed precision follows the usual rule of about 16 bytes per parameter before activations:

That is about 128 GB for an 8B model, which is why full SFT of anything above a few billion parameters needs sharding (ZeRO-3 or FSDP) across GPUs. Activation memory scales with batch tokens, layers and hidden size and is controlled with gradient checkpointing. LoRA drops the gradient and optimiser terms to the adapter size, giving roughly $2P$ plus activations, and QLoRA stores the frozen base in 4 bits for about $0.5P$. Relative to other stages: DPO costs about 2 to 4 times SFT per example (two responses, plus a reference forward pass unless log-probs are precomputed); PPO and GRPO cost one to two orders of magnitude more because generation dominates.

Failure modes and how to spot them#

Overfitting and memorisation

Training loss falls toward zero while held-out loss bottoms out and rises. Generations start reproducing training responses verbatim, or the model answers every prompt in the house style of one dominant data source. Track held-out NLL on a split of your SFT data and, more importantly, a generation eval on held-out prompts every epoch.

Catastrophic forgetting

The model gets better at the target task and worse at things you did not train on: general knowledge benchmarks, other languages, coding, safety refusals. Run a fixed regression suite of general benchmarks before and after. The standard mitigations are mixing in general instruction data, lowering the learning rate, training fewer epochs, or using LoRA, which Biderman et al. (2024) found forgets less than full fine-tuning while also learning less.

Format and stopping bugs

The model never stops, stops mid-sentence, emits template tokens as text, or produces a double beginning-of-sequence. These almost always trace back to a mismatch between the template used in training and at inference, an unsupervised end token, or truncation. Before any long run, decode a few tokenised training examples back to text and inspect exactly which tokens carry labels.

Teaching hallucination

If demonstrations contain facts the base model does not know, SFT teaches the model to produce confident statements about things it cannot verify, which generalises to confident fabrication elsewhere. Prefer demonstrations whose factual content the model could plausibly have produced, and let retrieval or tools supply the rest.

A toy run: overfitting and forgetting, with real numbers#

To see both main failure modes in one place, we ran a seeded numpy simulation with shared parameters, so that training on one set of contexts moves predictions on others the way it does in a real network. The model is a softmax over a 64-token vocabulary, ${\pi }_W(\cdot \mid c)=\mathrm{softmax}(W\varphi (c))$, with 32-dimensional context features $\varphi (c)$. The "pretrained" weights $W_0$ define the base model's behaviour on 200 general contexts. The task distribution is a different mapping $W_{\text{task}}=W_0+\Delta$. The SFT set has 200 task contexts with 8 sampled target tokens each (1,600 examples); 40 task contexts are held out. We trained with SGD (learning rate 4.0, batch 32, so 50 steps per epoch) for 2,000 steps and measured KL from the true task distribution on held-out contexts, and KL from the base model on the general contexts, which is our forgetting metric. We repeated the run with 10% and 25% of each batch replaced by samples from the base model on general contexts (rehearsal).

All three runs start at a held-out task KL of 1.273 nats. The held-out curve is U-shaped: it falls quickly, bottoms out around 20 epochs in the pure-SFT run, then rises as the model memorises the 8 noisy samples per context. Forgetting rises monotonically the whole time; it does not stop when task learning stops. Rehearsal trades task fit for retention: at their best checkpoints, 10% general data cut forgetting by 26% (0.762 to 0.566) at the cost of 0.074 nats of task KL. The 25% run forgot much less at its best step but also peaked earlier and lower. The general data in this toy is itself small (400 samples) so rehearsal eventually stops helping, which is also what happens with a tiny replay buffer in practice.

The practical reading: pick checkpoints on held-out generation quality, stop when it plateaus rather than when training loss does, and budget for a regression suite that would have caught the forgetting curve, because nothing in the task metrics shows it.

When to use SFT, and what to use instead#

Use SFT whenever you have, or can write, examples of the exact output you want: new formats, tool-call schemas, a house style, a narrow extraction or classification task, a base model that needs to learn to chat, or reasoning traces from a stronger model. It is the most data-efficient way to move a model into a new region of behaviour, and it is almost always the stage before preference tuning or RL. It is a poor tool when the quality you care about is easier to judge than to demonstrate (use preference data or a reward), when you need the model to avoid specific failure patterns (SFT has no negative signal), or when demonstrations come from a source whose outputs are far from what your model can produce (the model will imitate surface features and hallucinate the rest).

Minimal implementation in PyTorch#

The sketch below uses Hugging Face Transformers for the model and tokenizer and a plain PyTorch loop, so every piece of the objective is visible. It masks everything except assistant turns and normalises by the number of supervised tokens. Libraries such as TRL wrap the same logic with packing and distributed training; in POFE this is the default SFT recipe with the masking and normalisation choices exposed as settings.

import torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer, get_cosine_schedule_with_warmup

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16).cuda()
model.gradient_checkpointing_enable()

def encode(messages, max_len=4096):
    ids, labels = [], []
    for i, m in enumerate(messages):
        # render each turn with the chat template; keep only the new tokens for this turn
        prefix = tok.apply_chat_template(messages[:i], tokenize=True) if i else []
        full = tok.apply_chat_template(messages[: i + 1], tokenize=True)
        new = full[len(prefix):]
        ids += new
        labels += new if m["role"] == "assistant" else [-100] * len(new)
    return torch.tensor(ids[:max_len]), torch.tensor(labels[:max_len])

opt = torch.optim.AdamW(model.parameters(), lr=1e-5, weight_decay=0.0)
sched = get_cosine_schedule_with_warmup(opt, int(0.03 * steps), steps)

for step, batch in enumerate(loader):            # batch: padded ids, labels (-100 on pad), attn mask
    out = model(input_ids=batch["ids"].cuda(), attention_mask=batch["mask"].cuda())
    logits = out.logits[:, :-1].float()
    targets = batch["labels"][:, 1:].cuda()
    n_tok = (targets != -100).sum().clamp(min=1)  # use the global count under grad accumulation
    loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1),
                           ignore_index=-100, reduction="sum") / n_tok
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step(); sched.step(); opt.zero_grad(set_to_none=True)

Rendering turn by turn assumes the template is prefix-stable (rendering $k$ messages produces a prefix of rendering $k+1$). Most templates are, but some insert or strip content depending on the final turn, so check the decoded output once. Under gradient accumulation, compute the supervised-token count over all micro-batches of the step and divide by that, not by each micro-batch's own count.

Further reading#

Ouyang et al. (2022) is the canonical description of SFT as stage one of RLHF, including the epoch and overfitting observations. Wei et al. (2021) and Chung et al. (2022) cover instruction tuning at scale. Zhou et al. (2023) is the strongest argument for quality over quantity. Lambert et al. (2024) is the most complete open account of a modern SFT mixture and its ablations. Huerta-Enochian and Ko (2024) and Shi et al. (2024) examine prompt-loss weighting, Jain et al. (2023) is NEFTune, and Biderman et al. (2024) quantifies the learning versus forgetting trade-off between LoRA and full fine-tuning.

Frequently asked questions#

References#

1. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. (2022). Training language models to follow instructions with human feedback. arxiv.org/abs/2203.02155
2. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, et al. (2021). Finetuned Language Models Are Zero-Shot Learners. arxiv.org/abs/2109.01652
3. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, et al. (2022). Scaling Instruction-Finetuned Language Models. arxiv.org/abs/2210.11416
4. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions. arxiv.org/abs/2212.10560
5. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, et al. (2023). LIMA: Less Is More for Alignment. arxiv.org/abs/2305.11206
6. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arxiv.org/abs/2411.15124
7. Mathew Huerta-Enochian, Seung Yong Ko (2024). Instruction Fine-Tuning: Does Prompt Loss Matter?. arxiv.org/abs/2401.13586
8. Zhengyan Shi, Adam X. Yang, Bin Wu, Laurence Aitchison, et al. (2024). Instruction Tuning With Loss Over Instructions. arxiv.org/abs/2405.14394
9. Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, et al. (2023). NEFTune: Noisy Embeddings Improve Instruction Finetuning. arxiv.org/abs/2310.05914
10. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, et al. (2024). LoRA Learns Less and Forgets Less. arxiv.org/abs/2405.09673

Related reading