GRPO (Group Relative Policy Optimization) explained

Group Relative Policy Optimization (GRPO) is a reinforcement-learning algorithm for language models that samples a group of completions for each prompt, scores them, and uses each completion's reward relative to the rest of its group as the advantage in a PPO-style clipped update. There is no learned value network. In the post-training stack it sits after supervised fine-tuning (or, in "zero" recipes, directly on a base model) and is the default optimiser for reinforcement learning with verifiable rewards: maths answers that can be checked, code that must pass tests, SQL that must return the right rows. It is also used with learned reward models, which is how it was first introduced.

Where GRPO came from and what it fixed#

GRPO was introduced by Zhihong Shao, Peiyi Wang, Qihao Zhu and colleagues at DeepSeek in DeepSeekMath (Shao et al., 2024). The paper's main contribution was a 120B-token maths pre-training corpus, but its RL section proposed a modification of PPO (Schulman et al., 2017) aimed squarely at memory. Standard RLHF with PPO keeps four networks in play: the policy being trained, a frozen reference copy for the KL penalty, a reward model, and a critic (value function) that is usually initialised from the reward model and trained alongside the policy. The critic is as large as the policy, needs its own gradients and optimiser state, and is hard to train well for language, where a reward typically arrives only at the last token of a long completion.

GRPO's observation is that you do not need a learned estimate of the expected reward if you can just measure it. Sample $G$ completions for the same prompt, and their mean reward is a Monte Carlo estimate of the value of that prompt under the current policy. Subtracting it from each completion's reward gives an advantage; dividing by the group standard deviation puts every prompt on a common scale. In DeepSeekMath, GRPO trained on roughly 144K GSM8K and MATH questions lifted DeepSeekMath-Instruct 7B from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH. A year later DeepSeek-R1 (DeepSeek-AI, 2025) used GRPO with rule-based rewards to train R1-Zero directly from a base model, and that result is what made GRPO the reference algorithm for reasoning RL.

The objective, symbol by symbol#

For a prompt $q$ drawn from the training distribution, sample $G$ completions $o_1,\dots ,o_G$ from the policy that generated the batch, ${\pi }_{{\theta }_{\text{old}}}$. Each completion $o_i$ is a token sequence of length $|o_i|$ with tokens $o_{i,t}$. A reward function gives a scalar $r_i$ per completion. The per-token probability ratio between the policy being optimised, ${\pi }_{\theta }$, and the sampling policy is

With outcome rewards, every token of completion $i$ gets the same advantage, the reward standardised within its group:

The GRPO objective from Shao et al. (2024), which is maximised, is

Here $\epsilon$ is the clipping range (0.2 is standard), $\beta$ the KL coefficient, and ${\pi }_{\text{ref}}$ a frozen copy of the starting policy. The KL term is not computed exactly over the vocabulary. GRPO uses the per-token estimator known as $k_3$ from Schulman's note on approximating KL divergence:

It is non-negative for every sample and unbiased for the true KL when tokens come from ${\pi }_{\theta }$. Unlike PPO-RLHF, which folds a KL penalty into the reward, GRPO adds it directly to the loss so it does not contaminate the advantage.

Why a group mean is a valid baseline

The policy-gradient theorem says ${\nabla }_{\theta }\mathbb{E}[r]=\mathbb{E}[(r-b){\nabla }_{\theta }\log {\pi }_{\theta }(o\mid q)]$ for any baseline $b$ that does not depend on the sampled action, because $\mathbb{E}[{\nabla }_{\theta }\log {\pi }_{\theta }]=0$. The best simple baseline is the expected reward $V(q)$, which is what a PPO critic tries to learn. The group mean is an unbiased estimate of $V(q)$ built from fresh samples. Strictly, including $r_i$ itself in the mean introduces a small bias that shrinks the gradient by a factor of $(G-1)/G$; the leave-one-out baseline in RLOO (Ahmadian et al., 2024) removes it. In practice the difference is a constant rescaling absorbed by the learning rate.

The std division is the part that is not a baseline. It multiplies each prompt's gradient by $1/\mathrm{std}$, which for binary rewards with success fraction $\hat{p}$ is $1/\sqrt{\hat{p}(1-\hat{p})}$. That weights prompts the model almost always fails (or almost always solves) more heavily than prompts near 50%. Liu et al. (2025) call this a question-level difficulty bias and remove it in Dr. GRPO, together with the $1/|o_i|$ factor, which they show favours long wrong answers.

GRPO next to PPO, RLOO and DAPO

The differences are concentrated in two places: how the advantage is computed, and how token losses are averaged.

Read side by side, GRPO is PPO with the critic replaced by Monte Carlo, RLOO is GRPO without the std division or clipping, and DAPO is GRPO with four engineering fixes for long chain-of-thought training.

What the gradient looks like

Ignore clipping (it is inactive on the first update of each batch, where $\rho =1$) and set $\beta =0$. Then the gradient of the objective for one group is

Every token in a completion that beat its group average has its log-probability pushed up, every token in a below-average completion is pushed down, and the step per token is ${\hat{A}}_i/|o_i|$. Two consequences follow directly. Tokens shared by good and bad completions (the problem restatement, the formatting) receive opposing pushes that roughly cancel, so the net signal concentrates on where completions differ. And because ${\sum }_i{\hat{A}}_i=0$, a group in which every reward is equal contributes exactly nothing.

Worked example: one group by hand#

Take $G=8$ completions of a maths problem with a binary verifier. Two are correct: $r=(1,0,0,1,0,0,0,0)$. The mean is $0.25$. The population standard deviation is $\sqrt{0.25\times 0.75}=0.433$, so

The two correct completions get pushed up three times harder than each wrong one gets pushed down, and the advantages sum to zero ($2\times 1.732-6\times 0.577=0$). Many implementations compute the standard deviation with Bessel's correction, as torch.std does by default; then $\mathrm{std}=0.463$ and the advantages become $1.620$ and $-0.540$. Running these numbers in numpy reproduces both sets exactly.

Now suppose a correct completion is 200 tokens long and a wrong one 800 tokens. Under the $1/|o_i|$ aggregation the per-token weight is $1.732/200=0.0087$ for the correct answer and $-0.577/800=-0.00072$ for the wrong one. A long wrong answer is penalised per token about twelve times more gently than a short right answer is rewarded. That asymmetry is the length bias Dr. GRPO targets: when the model is wrong, being wrong at length is cheap.

Finally, the clip. Say after one inner gradient step the policy has raised a token in a correct completion from probability 0.50 to 0.65, so $\rho =1.3$. With $\hat{A}=1.732$ and $\epsilon =0.2$, the surrogate is $\min (1.3\times 1.732,1.2\times 1.732)=2.078$, the clipped branch, which has zero gradient. That token stops contributing until the next batch. If instead the token were in a wrong completion with $\hat{A}=-0.577$ and its probability had dropped to $\rho =0.7$, the surrogate $\min (0.7\times -0.577,0.8\times -0.577)=-0.462$ is again the clipped branch, and the push down stops.

The algorithm, step by step#

for step in range(num_steps):
    prompts = sample_batch(dataset, B)                       # B prompts
    with no_grad():
        groups = [policy_old.generate(q, n=G, temperature=1.0) for q in prompts]
        rewards = [[reward_fn(q, o) for o in g] for q, g in zip(prompts, groups)]
        adv = [(r - mean(r)) / (std(r) + 1e-4) for r in rewards]   # one scalar per completion
        logp_old = policy_old.token_logprobs(prompts, groups)
        logp_ref = ref.token_logprobs(prompts, groups) if beta > 0 else None
    for epoch in range(mu):                                  # mu = 1 is common
        for mb in minibatches(prompts, groups, adv, logp_old, logp_ref):
            logp = policy.token_logprobs(mb)
            ratio = exp(logp - mb.logp_old)
            surr = minimum(ratio * mb.adv, clip(ratio, 1 - eps, 1 + eps) * mb.adv)
            kl = exp(mb.logp_ref - logp) - (mb.logp_ref - logp) - 1 if beta > 0 else 0
            per_token = surr - beta * kl
            loss = -mean_over_sequences(mean_over_tokens(per_token, mb.mask))
            loss.backward(); optimizer.step(); optimizer.zero_grad()
    policy_old.load_state_dict(policy.state_dict())          # sync generator weights

The loop has two phases with very different costs. Generation produces $B\times G$ completions, typically with an inference engine such as vLLM holding a copy of the weights. Training then makes $\mu$ passes over those completions. With $\mu =1$ and a single minibatch the ratio is exactly 1, the clip never activates, and GRPO reduces to a REINFORCE estimator with a group baseline. Clipping only matters when you take several minibatch steps per batch, which is how most large runs amortise generation cost.

Data it needs

GRPO needs prompts, not completions. Each record is a prompt plus whatever the reward function needs to score an answer: a reference final answer for maths, unit tests for code, a schema and expected rows for SQL. A few thousand prompts is enough for a narrow task; reasoning runs use tens of thousands (DAPO-Math-17K has 17K, DeepSeekMath used about 144K). What matters more than count is the pass-rate distribution under the starting policy. Prompts the model solves 0% or 100% of the time produce zero-variance groups and contribute nothing; a useful set has a broad spread between, say, 10% and 90%. Measure it before training by sampling 8 to 16 completions per prompt.

Hyperparameters that matter#

Tune in this order: get the reward function right on a held-out sample, pick $G$ and temperature so that fewer than half the groups are zero-variance, then sweep the learning rate over a factor of 3 in each direction for a few hundred steps. Only then touch $\beta$ and $\epsilon$. Open-source trainers have also moved the defaults: TRL's GRPOTrainer now defaults to $\beta =0$ and token-level (DAPO-style) aggregation, citing Open-Reasoner-Zero (Hu et al., 2025) for the finding that the KL term is not essential. Check which loss your library is actually computing before comparing runs to a paper.

Compute and memory#

For full fine-tuning in mixed precision with Adam, the trainable policy costs about 16 bytes per parameter (bf16 weights and gradients plus fp32 master weights and two moments). A frozen bf16 reference adds 2 bytes per parameter. The generation engine holds another bf16 copy plus its KV cache. A rough rule for a policy with $N$ parameters:

For a 7B model that is roughly 112 GB of training state, 14 GB for the reference and 14 GB plus cache for generation, before activations, so a single 8-GPU node of 80 GB cards is comfortable. PPO with a same-size critic and reward model would add about 126 GB more. With LoRA the policy state shrinks to roughly the 2-byte frozen weights plus a small adapter, and the reference is free (disable the adapter to get it).

Time is dominated by generation. Each step generates $B\cdot G$ completions; at $B=128$, $G=8$ and a mean length of 1,000 tokens that is about a million tokens generated autoregressively per step, and those same million tokens then go through a forward pass for the old and reference log-probabilities and a forward and backward pass for the update. An SFT step on 128 examples of similar length processes an eighth as many tokens and never decodes. In practice a GRPO step costs several times an SFT step and a full GRPO run costs one to two orders of magnitude more than a DPO run on a comparable dataset, because DPO never samples.

A toy run: what the group baseline buys#

To see the mechanism without an LLM, we built a tabular model: 256 "prompts", each a softmax over 10 possible answers with one correct, rewarded $+1$ if correct and $-1$ otherwise (the convention DAPO uses). Logits start random so initial pass rates vary; the mean is 17.2%. The script is seeded and every number below is its output.

First, gradient quality on a single prompt with success probability $p=0.2$. We draw 20,000 groups and measure the cosine between each group's gradient estimate and the true gradient of expected reward (a scale-free measure, so the std division does not flatter GRPO).

Three things show up. At $G=8$ and $G=16$ the group baseline gives a better-aimed gradient than raw REINFORCE. At $G=4$ it is worse, because 40.8% of groups are all-fail and contribute a zero vector, while REINFORCE still learns something from pushing down sampled wrong answers. And the std division changes nothing about direction within one prompt: it is purely a per-prompt step size. We also checked the 0/1 reward convention: there REINFORCE without a baseline scored 0.834 at $G=8$, better than the group baseline, because failures carry zero reward and therefore no noise. Baselines matter most when rewards have an offset, which is the usual case once format bonuses or $\pm 1$ rewards enter.

Second, training. With 32 prompts per step, $G=8$, 80 steps, and the learning rate tuned separately per estimator over {0.3, 1, 3, 10}, the mean pass rate rose from 17.2% to 89.7% with no baseline, 93.8% with the group mean, and 94.2% with GRPO (5 seeds, standard deviation under 1.1 points). Adding an exact KL penalty to the initial policy at $\beta \in \{0,0.04,0.2,1.0\}$ gave final pass rates of 94.2%, 94.1%, 93.3% and 54.5%, with final KL of 2.02, 2.01, 1.97 and 0.82 nats. Small $\beta$ barely restrains a policy that is being pulled hard by a clean reward; it starts to bind only when it is comparable to the advantage scale. That matches why many verifiable-reward recipes drop it and why it earns its place when the reward is a learned model you are trying not to exploit.

Failure modes and how to spot them#

Zero-variance starvation. If most prompts are too hard or too easy, most groups have identical rewards and the effective batch shrinks. Log the fraction of groups with $\mathrm{std}(r)=0$ every step. Above 50% you are paying for rollouts that do nothing; filter prompts by pass rate or use DAPO's dynamic sampling, which resamples until the batch is full of informative groups.

Length drift. Mean completion length rising steadily while accuracy plateaus is the signature of the $1/|o_i|$ bias, especially on incorrect completions. Plot length separately for correct and incorrect answers; if the incorrect curve climbs faster, switch to a token-level or constant normaliser.

Entropy collapse. Token entropy falling fast in the first hundred steps, followed by a flat reward and a rising zero-variance rate, means the policy has become nearly deterministic and groups no longer explore. Raise the temperature, the upper clip bound, or $G$.

Reward hacking. Reward climbing while held-out accuracy does not. With verifiers this usually means a parser gap (answers that the regex accepts but a human would not); with reward models it is the familiar over-optimisation. Keep a held-out evaluation with a stricter grader and read samples every few hundred steps.

Off-policy instability. With $\mu >1$ or asynchronous generation, watch the clip fraction (share of tokens on the flat part of Figure 2) and the approximate KL between ${\pi }_{\theta }$ and ${\pi }_{{\theta }_{\text{old}}}$. A clip fraction above roughly 20% means the step is too big for the data you have.

Truncation noise. Completions cut off at the length limit score 0 regardless of quality. A rising truncation rate makes the reward curve fall for reasons unrelated to capability. Either mask truncated completions from the loss or raise the limit.

When to use GRPO, and what to use instead#

GRPO is the right default when you can score a completion automatically and you want the model to discover solutions better than any demonstration you could write. It is the wrong tool when you have no reliable reward, or when the behaviour you want is already present in data you could imitate.

Minimal implementation#

The core of GRPO is about thirty lines on top of a model that can return per-token log-probabilities. The sketch below assumes completions are already generated and padded, with a mask marking completion tokens.

import torch

def grpo_advantages(rewards: torch.Tensor, G: int, scale: bool = True, eps: float = 1e-4):
    """rewards: (B*G,) ordered prompt-major. Returns (B*G,) advantages."""
    r = rewards.view(-1, G)
    a = r - r.mean(dim=1, keepdim=True)
    if scale:
        a = a / (r.std(dim=1, keepdim=True) + eps)
    return a.view(-1)

def grpo_loss(logp, logp_old, adv, mask, logp_ref=None, beta=0.0, clip_eps=0.2):
    """logp, logp_old, logp_ref, mask: (N, T) per-token; adv: (N,)."""
    ratio = torch.exp(logp - logp_old)
    a = adv.unsqueeze(1)
    surr = torch.minimum(ratio * a, torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * a)
    if beta > 0 and logp_ref is not None:
        d = logp_ref - logp
        surr = surr - beta * (torch.exp(d) - d - 1)          # k3 estimator, per token
    per_seq = (surr * mask).sum(1) / mask.sum(1).clamp(min=1) # 1/|o_i| (original GRPO)
    return -per_seq.mean()                                   # 1/G over the batch

# Usage inside a training step:
# logits = policy(input_ids).logits[:, :-1]
# logp = torch.gather(logits.log_softmax(-1), 2, input_ids[:, 1:, None]).squeeze(-1)
# loss = grpo_loss(logp, logp_old, grpo_advantages(rewards, G), completion_mask)

For real runs, use a library that handles generation and weight syncing. In TRL the equivalent is a GRPOTrainer with a Python reward function:

from trl import GRPOConfig, GRPOTrainer

def correct(completions, answer, **kw):
    return [1.0 if extract_final(c) == a else 0.0 for c, a in zip(completions, answer)]

cfg = GRPOConfig(output_dir="out", num_generations=8, learning_rate=1e-6,
                 beta=0.04, epsilon=0.2, loss_type="grpo", max_completion_length=1024)
GRPOTrainer(model="your-sft-checkpoint", args=cfg, reward_funcs=correct,
            train_dataset=ds).train()

Setting loss_type="grpo" and beta=0.04 reproduces the DeepSeekMath formulation; the library defaults implement the later variants. In POFE, GRPO is a recipe: you supply the prompt set and the reward function, and the run is judged on the metrics above, not on the reward curve alone.

Further reading#

Shao et al. (2024) is the primary source for the objective and the KL estimator; DeepSeek-AI (2025) shows GRPO at scale on a base model. Liu et al. (2025) and Yu et al. (2025) are the two essential critiques and repairs, covered in our Dr. GRPO and DAPO articles. Ahmadian et al. (2024) is the best argument that much of PPO's machinery is unnecessary for language models in the first place.

Frequently asked questions#

References#

1. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arxiv.org/abs/2402.03300
2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948
3. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arxiv.org/abs/1707.06347
4. John Schulman (2020). Approximating KL Divergence. joschu.net/blog/kl-approx.html
5. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arxiv.org/abs/2503.20783
6. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arxiv.org/abs/2503.14476
7. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arxiv.org/abs/2402.14740
8. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum (2025). Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model. arxiv.org/abs/2503.24290
9. Hugging Face TRL maintainers (2025). TRL GRPO Trainer documentation. huggingface.co/docs/trl/main/en/grpo_trainer

Related reading