Open the workspace
Glossary

Group Relative Policy Optimization (GRPO)

GRPO is a PPO variant that uses the mean reward of a group of samples as its baseline instead of a value network. The advantage formula, why it saves memory, and where it fails.

POFE ResearchPublished 2026-09-05Updated 2026-10-304 min read

Group Relative Policy Optimization (GRPO): A reinforcement-learning method that scores a group of sampled completions per prompt and uses their mean reward as the baseline, removing PPO's value network.

Explained#

GRPO was introduced with DeepSeekMath (Shao et al., 2024). For each prompt it samples completions, scores them with a reward function or verifier, and standardises rewards within the group:

The update is PPO's clipped objective with this advantage and a KL penalty to a reference model. Dropping the learned value function roughly halves memory relative to PPO, which is why GRPO made RL practical for teams without large clusters.

Where it fails

  • Zero-variance groups. If every sample passes or every one fails, all advantages are zero. A prompt solved 5% of the time gives an all-fail group of 8 with probability .
  • Difficulty weighting. Dividing by the std up-weights near-0% and near-100% prompts; Dr. GRPO (Liu et al., 2025) removes it.
  • Reward hacking whenever the verifier has gaps.

Our group-size simulation quantifies the zero-variance waste and the coverage trade-off under a fixed rollout budget.

References#

  1. Shao, Z. et al. (2024). DeepSeekMath. arXiv:2402.03300. arxiv.org/abs/2402.03300
  2. Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783. arxiv.org/abs/2503.20783
GRPO group size under a fixed rollout budgetResearchReward hackingGlossaryDirect Preference Optimization (DPO)GlossarySFT vs DPO vs GRPO: choosing a post-training methodGuide

Bring a model and a metric.

Tell us what the model should get better at. A post-training engineer replies within one business day with a plan, a compute estimate and an eval design.