Post-training algorithms 101
One page per algorithm, grouped by family. Each covers the objective symbol by symbol, a worked example, the hyperparameters that matter, failure modes and a minimal implementation.
SFT and adapters
Supervised training and the parameter-efficient ways to run it.
Supervised fine-tuning (SFT), explained
Supervised fine-tuning trains a pretrained model with next-token cross-entropy on curated prompt and response demonstrations, usually counting the loss only on response tokens.
Read →Algorithms 101LoRA (low-rank adaptation), explained
LoRA freezes the pretrained weights and trains a low-rank additive update BA for selected matrices, cutting trainable parameters and optimiser memory by orders of magnitude.
Read →Algorithms 101QLoRA, explained
QLoRA trains bf16 LoRA adapters on top of a frozen base model stored in 4-bit NormalFloat, cutting base-weight memory to about half a byte per parameter.
Read →Algorithms 101DoRA (weight-decomposed low-rank adaptation), explained
DoRA reparameterises each weight as a trainable per-column magnitude times a unit-norm direction, fine-tuning the direction with LoRA and the magnitude directly.
Read →Algorithms 101LoRA+ (separate learning rates for A and B), explained
LoRA+ trains LoRA's output-side matrix B with a learning rate lambda times larger than the input-side matrix A, so both factors keep learning as models get wider.
Read →Algorithms 101PiSSA (principal singular values and vectors adaptation), explained
PiSSA puts the top-r singular components of each pretrained weight into LoRA's trainable factors and freezes the residual, so fine-tuning starts in the weight's most important directions.
Read →Algorithms 101GaLore (gradient low-rank projection), explained
GaLore keeps weights full rank but runs Adam on a low-rank projection of each gradient, refreshing the projection by SVD every few hundred steps.
Read →Algorithms 101Prompt tuning and prefix tuning, explained
Prompt tuning learns a few continuous token embeddings prepended to the input of a frozen model; prefix tuning learns key and value vectors prepended in every attention layer.
Read →Algorithms 101IA3 (infused adapters by inhibiting and amplifying activations), explained
IA3 learns three vectors per transformer block that rescale attention keys, values and MLP hidden activations, adapting a frozen model with about 0.01% extra parameters.
Read →Distillation and self-training
Learning from a stronger model, or from your own filtered samples.
Knowledge distillation for LLMs, explained
Knowledge distillation trains a smaller student model to imitate a larger teacher, either by fine-tuning on teacher outputs or by matching the teacher's next-token distributions.
Read →Algorithms 101On-policy distillation (GKD), explained
On-policy distillation trains a student on sequences it generates itself, minimising a per-token divergence from a teacher's distributions at the states the student actually visits.
Read →Algorithms 101Rejection-sampling fine-tuning, STaR and expert iteration, explained
Rejection-sampling fine-tuning samples several responses per prompt, keeps those a verifier or reward model accepts, fine-tunes on them with the SFT loss, and repeats.
Read →Algorithms 101SPIN explained: self-play fine-tuning for language models
An iterative method that trains a model to prefer the human-written SFT response over its own previous generation, using a DPO-form loss with the previous iteration as reference.
Read →Algorithms 101Self-rewarding language models, explained
An iterative alignment method in which the model generates candidate responses, scores them itself with an LLM-as-a-judge prompt, and trains with DPO on its best-versus-worst pairs, improving both as actor and as judge.
Read →Algorithms 101BOND (Best-of-N distillation), explained
BOND fine-tunes a policy to match the distribution of Best-of-N sampling, so one sample costs what one sample costs but behaves like the best of N.
Read →Reward models and classic RL
Learned rewards and the policy-gradient methods that optimise them.
Reward modeling with Bradley-Terry
A reward model maps a prompt and response to a scalar score, trained on human preference pairs with the Bradley-Terry logistic loss on score differences.
Read →Algorithms 101Process reward models
A process reward model scores every step of a multi-step solution, giving step-level feedback for reranking, search and dense RL rewards instead of one outcome score.
Read →Algorithms 101RLHF with PPO
RLHF with PPO samples responses from the policy, scores them with a reward model, and updates the policy with a clipped policy-gradient objective under a KL penalty to the SFT model.
Read →Algorithms 101RLAIF and Constitutional AI
RLAIF trains on preference labels produced by a language model instead of humans; Constitutional AI steers that judge with written principles and adds a critique-and-revise SFT stage.
Read →Algorithms 101REINFORCE and RLOO
RLOO is REINFORCE with k samples per prompt, where each sample's baseline is the mean reward of the other k-1, giving an unbiased low-variance policy gradient with no value network.
Read →Algorithms 101REINFORCE++ explained
A critic-free PPO-style method that normalises advantages over the whole training batch instead of each prompt's group, in a single-sample variant for RLHF and a group-mean variant for reasoning tasks.
Read →Algorithms 101ReMax
ReMax is REINFORCE for LLMs with the reward of the policy's own greedy response as the baseline, removing PPO's value model and most of its hyperparameters.
Read →Offline preference methods
Training directly on chosen and rejected responses, no RL loop.
DPO (Direct Preference Optimization), explained
DPO trains a language model directly on chosen/rejected response pairs with a logistic loss on the implicit reward margin, optimizing the RLHF objective without a reward model or RL loop.
Read →Algorithms 101IPO (Identity Preference Optimization), explained
IPO is a DPO-style preference loss that regresses the chosen-versus-rejected log-likelihood-ratio gap onto a fixed target 1/(2τ), so the KL regularization stays effective even when preferences are deterministic.
Read →Algorithms 101KTO (Kahneman-Tversky Optimization), explained
KTO trains a language model from single responses labeled desirable or undesirable, scoring each response's implied reward against a KL reference point with a prospect-theory-style sigmoid value function.
Read →Algorithms 101ORPO (Odds Ratio Preference Optimization), explained
ORPO trains a base model in a single stage with the usual SFT loss on chosen responses plus a λ-weighted penalty on the log odds ratio of chosen versus rejected responses, with no reference model.
Read →Algorithms 101SimPO (Simple Preference Optimization), explained
SimPO is a reference-free DPO variant that uses the policy's average per-token log-likelihood as the reward and requires the chosen response to beat the rejected one by a target margin γ.
Read →Algorithms 101CPO (Contrastive Preference Optimization), explained
CPO trains on preference pairs with a reference-free DPO-style loss (uniform reference) plus a negative log-likelihood term on the preferred response, letting a model learn past the quality of its reference data.
Read →Algorithms 101Online and iterative DPO, explained
Online and iterative DPO apply the DPO loss to preference pairs sampled from the current policy and labeled by a reward model, judge or verifier, either every step or in rounds, so training data stays on-policy.
Read →Algorithms 101Step-DPO (step-level preference optimization), explained
Step-DPO trains a model to prefer a correct next reasoning step over the first wrong one, given a shared correct prefix, using the DPO loss on step spans only.
Read →Algorithms 101SLiC-HF (Sequence Likelihood Calibration with Human Feedback) explained
An offline alignment method that trains a model so the log-likelihood of preferred responses exceeds that of rejected ones by a margin, with a cross-entropy term that keeps it near its SFT behaviour.
Read →Algorithms 101RRHF (Rank Responses to align Human Feedback) explained
An offline alignment method that trains a model so its length-normalised log-likelihood ranks k responses in reward order, plus supervised fine-tuning on the highest-reward response.
Read →Algorithms 101SPPO and Nash learning from human feedback, explained
Methods that treat alignment as a two-player preference game: NLHF defines the Nash-equilibrium target and the Nash-MD algorithm, and SPPO approximates it by regressing log-probability ratios onto self-play win rates.
Read →RL for reasoning
Group baselines and verifiable rewards for maths, code and agents.
GRPO (Group Relative Policy Optimization) explained
A PPO-style RL algorithm that samples a group of completions per prompt and uses their reward relative to the group mean as the advantage, removing the value network.
Read →Algorithms 101Dr. GRPO explained: GRPO without the length and difficulty bias
A corrected form of GRPO that drops per-response length normalisation and group std scaling, giving an unbiased advantage and stopping incorrect answers from growing longer.
Read →Algorithms 101DAPO explained: decoupled clip and dynamic sampling policy optimisation
A GRPO-based RL recipe for long reasoning that decouples the clip range, filters out zero-signal groups, averages the loss over tokens and softly penalises overlong responses.
Read →Algorithms 101GSPO (Group Sequence Policy Optimization) explained
A GRPO variant from the Qwen team that computes one length-normalised importance ratio per response and clips whole responses, which stabilises off-policy RL training, especially for MoE models.
Read →Algorithms 101VAPO and value-based RL for long chain-of-thought, explained
A PPO recipe from ByteDance Seed that makes a learned value function work for long chain-of-thought through value pretraining, decoupled and length-adaptive GAE, plus DAPO-style loss tweaks; VinePPO instead estimates values with Monte Carlo rollouts.
Read →Algorithms 101RLVR explained: reinforcement learning with verifiable rewards
Reinforcement learning in which the reward is computed by a program that checks the output against ground truth, such as an answer checker or unit tests, instead of a learned reward model.
Read →Model merging
Combining trained models without more training.
Bring a model and a metric.
Tell us what the model should get better at. A post-training engineer replies within one business day with a plan, a compute estimate and an eval design.