GRPO group size under a fixed rollout budget

Group Relative Policy Optimization replaces PPO's learned value baseline with the mean reward of a group of $G$ samples for the same prompt. That one design choice fixes memory, but it creates a budgeting question every team running RL with verifiable rewards has to answer: for a fixed number of rollouts, should you sample many completions for few prompts, or few completions for many prompts? We built a small, fully reproducible simulation to isolate that trade-off, and to look at the standard-deviation normalisation that recent work argues is a source of bias.

The GRPO advantage, and where it vanishes#

For a prompt $q$, GRPO samples $G$ completions $o_1,\dots ,o_G$ from the current policy and scores each with a reward $r_i$. The advantage of completion $i$ is its reward standardised within the group (Shao et al., 2024):

The policy gradient is a sum of ${\hat{A}}_i{\nabla }_{\theta }\log {\pi }_{\theta }(o_i\mid q)$ terms, clipped and KL-regularised as in PPO. With binary verifiable rewards — the SQL ran and returned the right rows, the unit tests passed — there is an obvious failure mode. If every completion in the group fails, or every one succeeds, all rewards are equal, the numerator is zero for every $i$, and the prompt contributes nothing to the update.

If the policy solves prompt $q$ with probability $p$, the chance that a group of $G$ independent samples is uninformative is

For a prompt the model solves 5% of the time, a group of 8 is all-fail with probability $0.95^8\approx 0.66$. Two thirds of the rollouts you pay for on that prompt are, for the gradient, wasted. Larger groups help — at $G=32$ it falls to 0.19 — but each of those groups costs four times as much.

The same function as a surface over pass rate and group size makes the trade-off visible: the "valley" of informative groups widens as G grows, but every step along the G axis doubles the cost of a group.

Simulation setup#

We wanted the simplest model in which group size has both of its real effects: more samples per prompt (fewer zero-variance groups, lower-variance advantages) and fewer prompts per step (less coverage). The model is deliberately tabular.

• Prompts. 2,000 prompts. Prompt $j$ has its own success logit $z_j$; the policy solves it with probability $\sigma (z_j)$. Initial pass rates are drawn from a Beta distribution: a hard set (Beta(0.6, 2.4), mean ≈ 20%) and a mixed set (Beta(1, 1), mean ≈ 50%).
• Budget. Every step spends exactly 512 rollouts: $512/G$ prompts are sampled without replacement and each gets $G$ rollouts. Every run spends 204,800 rollouts in total.
• Update. For a Bernoulli policy, $\partial \log \pi (r)/\partial z=r-p$, so the per-prompt update is $z_j\leftarrow z_j+\eta \cdot \frac{1}{G}{\sum }_i{\hat{A}}_i(r_i-p_j)$ with $\eta =0.5$.
• Variants. With std normalisation (as above) and without it (${\hat{A}}_i=r_i-\bar{r}$), the change proposed in Dr. GRPO (Liu et al., 2025).
• Seeds. 8 seeds per cell; standard deviations of the final pass rate across seeds are below 0.8 points everywhere, so the differences below are not noise.

Results#

Under a fixed budget, smaller groups won in every configuration we ran. On the hard set with std normalisation, $G=2$ lifted the mean pass rate from 19.9% to 65.1%, while $G=64$ reached only 24.1% for the same number of rollouts.

Hard set, std-normalised. "Zero-variance groups" is the fraction of all groups sampled during training whose rewards were all equal.

Two things in the table are worth pausing on. First, the zero-variance fraction at $G=2$ is 78.5% — most pairs carry no signal — and $G=2$ still wins. Wasted groups are cheap when groups are small, and coverage matters more: at $G=2$ every prompt is visited roughly 51 times during training; at $G=64$, fewer than twice. Second, the hard prompts are where the gap is largest: 29.0% versus 4.3%.

What std normalisation is doing#

Removing the std term lowered the final pass rate everywhere in this model — on the hard set at $G=8$, from 49.7% to 33.6%. It is tempting to read that as "std normalisation is good". The mechanism says something more specific.

With binary rewards and $k$ successes in a group of $G$, the group std is $\sqrt{\hat{p}(1-\hat{p})}$ with $\hat{p}=k/G$. A single success in a group of 8 gets a normalised advantage of

Dividing by the std multiplies the update for rare successes on hard prompts — and rare failures on easy ones — by up to about 3×. In a tabular model, where a bigger step on a hard prompt is pure gain, that looks like an improvement. It is the same effect Liu et al. (2025) describe as a question-level difficulty bias: prompts are weighted by how close to 0% or 100% the policy is, not by how much they matter. In a real model with shared parameters, that reweighting changes what the whole network learns, and they report that removing it (along with the per-response length normalisation) improves token efficiency without losing accuracy.

Our simulation can't adjudicate that debate, because it has no shared parameters. It does show the size of the effect: normalisation is roughly a 2–3× learning-rate multiplier concentrated on the hardest and easiest prompts.

What to do with this in a real run#

1. Measure your zero-variance rate. Log the fraction of groups with identical rewards every step. If it is above ~50%, you are paying for rollouts that do nothing. POFE plots it next to the reward curve by default.
2. Filter prompts by current pass rate. Prompts the policy always solves or never solves contribute little. Dynamic sampling — resample or skip groups with zero variance, as in DAPO (Yu et al., 2025) — keeps the batch full of informative groups.
3. Don't default to large groups. 8–16 is a reasonable starting point for LLMs because cross-prompt generalisation rewards lower-variance advantages, but our results show the coverage cost is real. If you have many diverse prompts, try a smaller $G$ with more prompts per step before buying more rollouts.
4. Treat std normalisation as a weighting choice. If hard prompts are the ones you care about, it helps. If response length is drifting upward or easy prompts dominate, try the Dr. GRPO variant.

Read the methods background in SFT vs DPO vs GRPO, the glossary entries for GRPO and reward hacking, or try a GRPO run in the workspace demo.

Frequently asked questions#

References#

1. Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. arxiv.org/abs/2402.03300
2. Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783. arxiv.org/abs/2503.20783
3. Yu, Q. et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476. arxiv.org/abs/2503.14476
4. Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. arxiv.org/abs/1707.06347

Related reading