Dr. GRPO explained: GRPO without the length and difficulty bias

Dr. GRPO ("GRPO Done Right") is a two-line change to Group Relative Policy Optimization that removes two normalisation terms: the division of each completion's loss by its own length, and the division of each group's advantages by the group's reward standard deviation. What is left is the PPO clipped objective with an unbiased Monte Carlo advantage, $r_i-\mathrm{mean}(r)$, and a fixed constant in place of the per-sequence length. It sits in exactly the same place as GRPO in the post-training stack, as the optimiser for RL with verifiable rewards on a base or SFT model, and it costs nothing extra to run. Its purpose is to stop the policy from learning that wrong answers should be long.

Where it comes from#

Dr. GRPO was proposed by Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee and Min Lin (Sea AI Lab, National University of Singapore and Singapore Management University) in Understanding R1-Zero-Like Training: A Critical Perspective, first posted in March 2025 and published at ICML 2025. The paper is an audit of the "R1-Zero" recipe made famous by DeepSeek-R1 (DeepSeek-AI, 2025): take a base model, apply GRPO with a rule-based correctness reward, and watch accuracy and response length climb together. That co-movement was widely read as the model learning to think longer. Liu et al. asked two questions. How much of what RL "discovers" was already in the base model? And how much of the length growth is produced by the optimiser rather than by the task?

Their answer to the second question is the algorithm. They show that GRPO's objective, as written in DeepSeekMath (Shao et al., 2024), contains two biases that the policy-gradient theorem does not call for. They also point out that the first one is not unique to GRPO: the PPO objective itself has no per-sequence length normalisation, yet many widely used open-source PPO implementations (they list TRL, OpenRLHF, verl, SimpleRL-Zero and Open-Reasoner-Zero) average the loss over each response's tokens anyway. It is a habit inherited from language-model pre-training that is harmless for next-token prediction and not harmless for RL.

The two biases#

Start from the GRPO objective. For a prompt $q$, sample $G$ completions $o_i$ of lengths $|o_i|$ from ${\pi }_{{\theta }_{\text{old}}}$, score them with rewards $r_i$, and let ${\rho }_{i,t}$ be the per-token ratio ${\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})/{\pi }_{{\theta }_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})$. With the KL term omitted, as in most verifiable-reward recipes,

The two boxed terms are what Dr. GRPO removes.

Response-level length bias

On the first gradient step for a batch ($\rho =1$, clipping inactive), the gradient contributed by completion $i$ is

Each token's weight is ${\hat{A}}_i/|o_i|$. When ${\hat{A}}_i>0$, shorter correct responses receive larger per-token updates, so the policy is pushed toward brevity when it is right. When ${\hat{A}}_i<0$, longer incorrect responses receive smaller per-token penalties, so the policy is pushed less hard away from long failures than from short ones. The net effect, compounded over thousands of steps, is that incorrect responses get longer. That is a property of the estimator, not of the task: the true policy gradient ${\nabla }_{\theta }\mathbb{E}[r]$ contains no $1/|o|$ factor.

Question-level difficulty bias

For binary rewards with $k$ successes in a group of $G$, write $\hat{p}=k/G$. The group std is $\sqrt{\hat{p}(1-\hat{p})}$, so the std division multiplies the whole group's gradient by $1/\sqrt{\hat{p}(1-\hat{p})}$. The exact gradient of the expected reward on a prompt with true pass rate $p$, parameterised by a logit, has magnitude proportional to $p(1-p)$: it is largest at 50% and vanishes at 0% and 100%. GRPO's scaling turns that profile into roughly $\sqrt{p(1-p)}$, flattening it and up-weighting prompts that are nearly always solved or nearly always failed. Liu et al. call this a question-level difficulty bias. It is a weighting choice disguised as a normalisation: nothing in the RL objective says a 1-in-8 prompt deserves a larger step than its share of the expected reward.

The Dr. GRPO objective#

Remove both terms and replace the per-sequence length with a constant $L_{\text{norm}}$ (the paper's code uses the generation budget, the maximum number of tokens):

Because $L_{\text{norm}}$ is the same for every sequence and every batch, it is just a learning-rate scale. The gradient at $\rho =1$ becomes

which is, up to the constant and a $(G-1)/G$ factor, the REINFORCE estimator with a leave-one-out baseline used by RLOO. Liu et al. describe it as recovering the original PPO objective with an unbiased Monte Carlo advantage. In other words, Dr. GRPO does not add anything. It subtracts the parts that were never justified by the derivation.

Side by side with its siblings

DAPO's token-level loss, posted eight days earlier, attacks the same length problem differently. Dividing by the total token count of the batch makes every token count equally, so long sequences are no longer down-weighted per token. But the denominator still changes with the batch's lengths, so the effective step size shrinks when responses get long. Dr. GRPO's constant denominator avoids that coupling. DAPO keeps the std division; Dr. GRPO drops it.

Worked example by hand#

Length. A group of $G=8$ with two correct answers has GRPO advantages $+1.732$ and $-0.577$ (population std 0.433). Take a correct answer of 200 tokens and a wrong one of 800. GRPO's per-token weights are $1.732/200=0.00866$ and $-0.577/800=-0.00072$: each token of the long wrong answer is pushed down twelve times more gently than each token of the short right one is pushed up. Under Dr. GRPO with $L_{\text{norm}}=2000$, the advantages are $r_i-\bar{r}=+0.75$ and $-0.25$, and the per-token weights are $0.75/2000=0.000375$ and $-0.25/2000=-0.000125$ regardless of length. The 800-token wrong answer now receives four times the total push-down of a 200-token wrong answer, because it has four times as many tokens that were each part of a failure.

Difficulty. For $G=8$ and $k$ successes, the total advantage mass in a group, ${\sum }_i|{\hat{A}}_i|$, is:

Relative to a 50/50 prompt, a 1-in-8 prompt gets 44% of the update under Dr. GRPO and 66% under GRPO. The two methods agree on the shape; GRPO flattens it by a factor of up to 1.5 at $G=8$, and the factor grows with $G$ because more extreme values of $\hat{p}$ become possible (at $G=64$ a single success gets a std of 0.124 and a weight of 8.1).

Toy run: a policy that can choose how long to ramble#

To see the length bias act over a training run, we built a two-stage token-level policy with shared parameters. Each of 64 prompts has its own logit ${\psi }_j$ for picking the right approach on the first token. After that, the policy emits tokens until it samples a stop token, with stop probability $\sigma ({\varphi }_c)$ on the right approach and $\sigma ({\varphi }_w)$ on a wrong one; ${\varphi }_c$ and ${\varphi }_w$ are shared across prompts. Initial expected lengths are 298 tokens on the right approach and 618 on a wrong one (after truncation at 1,500). A response is correct if it took the right approach, ran at least 100 tokens (a minimum amount of work), and stopped before the 1,500-token cap. Each step samples 16 prompts with $G=8$; exact per-token score functions are summed according to each variant's normalisation; updates use Adam at learning rate 0.02 for 400 steps; 6 seeds. Every number below is this script's output.

The mechanism matches the analysis. With $1/|o_i|$ in the loss, wrong responses get longer every step, because the per-token push away from a failure is weaker the longer the failure is, and the policy discovers that dragging on is the cheapest way to be wrong. The same weighting pushes short "right approach but under 100 tokens" failures up hard, which grows correct-branch length too, until truncation at the cap starts turning right answers into zeros. The std term made no measurable difference to length in this toy (27.4% versus 27.3% accuracy, 1,410 versus 1,405 tokens). That is consistent with Liu et al.'s attribution: the length growth comes from the per-response normalisation.

What the paper found#

Liu et al. trained Qwen2.5-Math-7B with Dr. GRPO on MATH questions of levels 3 to 5, using the Qwen-Math prompt template, $G=8$, learning rate $10^{-6}$ (constant), clip $\epsilon =0.2$, KL coefficient 0, temperature 1.0 and a 3,000-token response budget. The run took 27 hours on 8 A100 GPUs. The resulting model, Oat-Zero-7B, scored 43.3% on AIME 2024, 62.7% on AMC, 80.0% on MATH500, 30.1% on Minerva Math and 41.0% on OlympiadBench, an average of 51.4%, which they report as state of the art for 7B zero-RL models at the time. Compared with GRPO under the same setup, reward curves were similar but response length stopped growing without bound, and incorrect responses on evaluation sets were substantially shorter.

The paper's findings about base models matter as much as the algorithm for anyone running these recipes. Qwen2.5 base models answer maths questions well with no prompt template at all, which the authors interpret as evidence of pre-training on concatenated question-answer text; reported RL gains on Qwen are therefore partly the model learning to answer in a format it already knew how to use. Self-reflection ("aha moment") behaviours already appear in DeepSeek-V3-Base before any RL. And with a template that matches the base model, a small and easier question set reached nearly the same accuracy as a large, harder one, suggesting that RL here mostly reinforces reasoning the model already has. When you evaluate any RLVR change, Dr. GRPO included, compare against the base model with a good template, not against the base model with a bad one.

The algorithm, step by step#

The loop is GRPO's loop with two lines changed:

for step in range(num_steps):
    prompts = sample_batch(dataset, B)
    groups  = [policy_old.generate(q, n=G, max_new_tokens=L_max) for q in prompts]
    rewards = tensor([[verify(q, o) for o in g] for q, g in zip(prompts, groups)])  # (B, G)
    adv = rewards - rewards.mean(dim=1, keepdim=True)        # change 1: no / std
    for mb in minibatches(prompts, groups, adv):
        ratio = exp(policy.logp(mb) - mb.logp_old)            # (N, T)
        surr  = minimum(ratio * mb.adv[:, None], clip(ratio, 1 - eps, 1 + eps) * mb.adv[:, None])
        loss  = -(surr * mb.mask).sum() / (mb.num_seqs * L_max)  # change 2: constant, not |o_i|
        loss.backward(); opt.step(); opt.zero_grad()
    policy_old.load_state_dict(policy.state_dict())

Data requirements are the same as GRPO: prompts with a way to verify answers, a few thousand to a few tens of thousands of them, filtered so the starting policy's pass rate is spread between 0% and 100%. Everything about generation, verifiers and infrastructure carries over unchanged.

Hyperparameters#

The one tuning trap: GRPO's std division silently multiplies the effective learning rate by 2 to 3 on typical binary-reward batches (a std of 0.5 doubles it; rarer successes triple it), and the length denominator changes from a few hundred tokens to the generation budget. A run that was stable with GRPO at $10^{-6}$ can be under-stepped with Dr. GRPO at the same nominal rate. Compare the gradient norm of the two in the first few steps and adjust.

Compute and memory#

Identical to GRPO. The changes touch two scalar operations per group, so memory is still about 16 bytes per trainable parameter for full fine-tuning with Adam, plus a generation copy and its KV cache, plus a reference model only if $\beta >0$. The practical saving is downstream: if incorrect responses stop growing, average rollout length falls, and rollout generation is where most of the wall-clock goes. A run whose mean length would have doubled is a run whose later steps cost nearly half as much.

Failure modes and what to watch#

Length still drifting. If mean length keeps rising under Dr. GRPO, check your library's aggregation. Many trainers expose several loss types and the default may be token-level DAPO or the original GRPO. Plot length separately for correct and incorrect responses; under Dr. GRPO the incorrect curve should not outgrow the correct one.

Hard prompts not moving. Without std scaling, a 1-in-16 success gets an advantage of 0.94, not 3.9. If your data is mostly hard, learning on those prompts is slower than under GRPO. That is the intended unbiased weighting, but if hard prompts are what you care about, consider curriculum or filtering rather than reintroducing the bias.

Reward scale sensitivity. The advantage now carries the reward's units. Mixing a 0/1 correctness reward with a 0 to 10 style score will let the larger one dominate. Normalise reward components before summing, or use batch-level scaling, which keeps the difficulty profile across prompts intact.

Zero-variance groups are unaffected by either fix: identical rewards still give zero advantage. Track their fraction as with GRPO.

When to use it#

Implementation#

import torch

def dr_grpo_loss(logp, logp_old, rewards, mask, G, L_norm, clip_eps=0.2):
    """logp, logp_old, mask: (B*G, T); rewards: (B*G,) prompt-major."""
    r = rewards.view(-1, G)
    adv = (r - r.mean(dim=1, keepdim=True)).view(-1, 1)    # no std division
    ratio = torch.exp(logp - logp_old)
    surr = torch.minimum(ratio * adv, torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * adv)
    n_seq = logp.shape[0]
    return -(surr * mask).sum() / (n_seq * L_norm)          # constant, not per-sequence length

In TRL, the same objective is two settings on a GRPOConfig: loss_type="dr_grpo", which normalises by a global constant equal to max_completion_length, and scale_rewards=False, which drops the std division. Setting only one of them gives you half of Dr. GRPO. The authors' own implementation lives in their open-source oat framework and the understand-r1-zero repository. Whichever library you use, plot response length split by correctness; it is the cheapest way to see the bias when it appears, and it is how we review GRPO runs in POFE.

Further reading#

Liu et al. (2025) is the source for everything specific here, including the audit of base models and templates. Read it alongside Shao et al. (2024) for the original objective and Yu et al. (2025) for DAPO's token-level alternative. Ahmadian et al. (2024) gives the REINFORCE-with-baseline view that Dr. GRPO converges to.

Frequently asked questions#

References#

1. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arxiv.org/abs/2503.20783
2. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arxiv.org/abs/2402.03300
3. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948
4. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arxiv.org/abs/2503.14476
5. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arxiv.org/abs/2402.14740
6. Sea AI Lab (2025). understand-r1-zero (code for the Dr. GRPO paper). github.com/sail-sg/understand-r1-zero
7. Hugging Face TRL maintainers (2025). TRL GRPOConfig source (loss_type and scale_rewards options). github.com/huggingface/trl/blob/main/trl/trainer/grpo_config.py

Related reading