REINFORCE++ explained

REINFORCE++ is a critic-free policy-gradient method for language models that keeps PPO's machinery (clipped ratio, minibatch updates, KL control) but replaces the learned value function with a single statistical step: normalise the advantages across the whole training batch rather than within each prompt's group. It comes in two forms. Plain REINFORCE++ samples one response per prompt and folds a KL penalty into the reward, which suits general RLHF against a reward model. REINFORCE++ with baseline samples several responses per prompt, subtracts the group mean, then divides by the batch-wide standard deviation, and adds a separate KL loss; it targets reasoning and agent tasks with verifiable rewards. Both sit where GRPO, RLOO and PPO sit, after SFT, and both ship as advantage estimators in OpenRLHF and verl.

Origin and the problem it targets#

REINFORCE++ was introduced by Jian Hu, with Jason Klein Liu, Haotian Xu and Wei Shen, in a paper first posted in January 2025 as REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models and revised several times; the current version (v9, November 2025) is titled REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization. Hu is also the lead author of OpenRLHF, where the method was first implemented, and the paper should be read with that context: it is as much a set of implementation choices distilled from running RLHF at scale as it is a new estimator.

The starting point is the family of methods that dropped PPO's critic: ReMax (a greedy-decoding baseline), RLOO (leave-one-out mean) and GRPO (group mean and standard deviation). The REINFORCE++ paper argues that the last of these has three problems, all caused by computing statistics from a handful of responses to one prompt. The group-standardised advantage is biased, because the numerator and the denominator are computed from the same few rewards and are not independent. It is unstable, because when a group's rewards happen to be nearly equal the standard deviation is tiny and the advantages blow up to full size. And it overfits, because the policy is rewarded for beating its siblings on the same prompt rather than for being good. The fix is to use statistics from the whole batch, typically a thousand responses or more, which are close to constants.

The empirical support is mixed in scale but consistent in direction. On general RLHF (Llama-3-8B-SFT, a Bradley-Terry reward model trained on about 700K preference pairs, 20K prompts), REINFORCE++ with one sample per prompt scored 46.7 on Arena-Hard versus GRPO's 46.8 with four samples, with shorter responses and a slower-rising KL. Trained on just 30 AIME-24 questions, GRPO reached 95.0% on the training set and 0.0 pass@1 on AIME-25, while REINFORCE++ reached 71.0% and 2.5 pass@1 (40.0 pass@16). On a multi-turn tool-use benchmark suite, REINFORCE++ with baseline averaged 24.10 versus 22.58 for GRPO and 21.85 for PPO. The paper also cites later independent ablations, including ScaleRL (Khatri et al., 2025), which found batch-level normalisation slightly better than prompt-level in compute efficiency and final performance.

The objective and the two advantage estimators#

Both variants optimise PPO's clipped surrogate. For prompt $q$, a response $o$ sampled from ${\pi }_{{\theta }_{\text{old}}}$, and token ratio $s_t(\theta )={\pi }_{\theta }(o_t\mid q,o_{<t})/{\pi }_{{\theta }_{\text{old}}}(o_t\mid q,o_{<t})$:

Everything that distinguishes REINFORCE++ is in $A_{q,o_t}$.

REINFORCE++ (one sample per prompt)

The advantage is the outcome reward minus the KL penalty accumulated from token $t$ to the end, the standard PPO-for-RLHF reward shaping with $\gamma =\lambda =1$:

Then the whole batch ${\mathcal{D}}_{\text{batch}}$ is standardised at once:

The batch mean is the baseline. It is a weak baseline, because it does not know how hard each prompt is, but it costs nothing and with one sample per prompt the batch covers as many distinct prompts as possible.

REINFORCE++ with baseline (k samples per prompt)

Sample $k>1$ responses per prompt. Subtract the group mean first, which removes prompt difficulty and makes the method indifferent to whether rewards are $\{0,1\}$ or $\{-1,1\}$; then standardise with batch statistics:

The KL is no longer in the reward. It becomes a separate loss term using the $k_2$ estimator:

Side by side with its neighbours, for response $i$ of prompt $q$ with group rewards $r_1,\dots ,r_k$:

REINFORCE++ with baseline is literally GRPO with the denominator swapped from a per-group to a per-batch statistic, plus a different KL estimator. The paper also describes it as PPO with the critic removed, GAE set to $\gamma =\lambda =1$, and the value baseline replaced by group-mean subtraction followed by batch normalisation.

Why the local standard deviation is biased

The paper's argument (its Appendix A) models a group's rewards as $r_i=\mu +{ϵ}_i$ with independent ${ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)$, and the GRPO advantage as $A_i=({ϵ}_i-\overline{ϵ})/D$ with $D$ the group's population standard deviation. Conditional on response $i$'s own noise,

so the denominator grows with $|{ϵ}_i|$: the more a response stands out, the more it inflates the very standard deviation it is divided by. That makes $\mathbb{E}[A_i\mid {ϵ}_i]$ a nonlinear, compressed function of ${ϵ}_i$ rather than a scaled copy of it. With a batch of a thousand responses, the batch standard deviation barely depends on any one of them and the distortion vanishes. The more practical problem is the other end. A group whose rewards differ only by reward-model noise gets standardised to unit scale anyway, so noise is presented to the optimiser as if it were signal. Our toy run below shows that this is the effect that matters.

Which KL estimator, and why it matters as a loss

All three estimators come from Schulman's note on approximating KL. With samples $y\sim {\pi }_{\theta }$ and $\delta ={\pi }_{\text{ref}}(y)/{\pi }_{\theta }(y)$: $k_1=-\log \delta$, $k_2=\frac{1}{2}(\log \delta {)}^2$, $k_3=\delta -1-\log \delta$. As estimators of the value of $\mathrm{KL}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$, $k_1$ and $k_3$ are unbiased. But when you differentiate them as a loss, holding the sampled $y$ fixed as every LLM trainer does, you get different gradients:

So $k_1$ as a loss does nothing in expectation (it belongs inside the reward, where REINFORCE++ puts it), $k_2$ gives exactly the reverse-KL gradient, and $k_3$, GRPO's choice, gives the forward-KL gradient, with an importance weight ${\pi }_{\text{ref}}/{\pi }_{\theta }$ that explodes when the policy has pushed a token the reference likes towards zero. We verified all three on a 5-token categorical distribution with exact expectations: $k_2$'s expected gradient matched the reverse-KL gradient with cosine 1.0000 and norm ratio 1.0000 at every distance tested; $k_3$'s matched the forward-KL gradient exactly and the reverse one only approximately (cosine 0.995 to 0.965, norm ratio up to 1.84 when the policy was far from the reference); $k_1$'s was zero to machine precision ($10^{-17}$). Far from the reference, the single-sample variance of the $k_3$ gradient was 3.83 against 0.51 for $k_2$.

Worked example: three normalisations of the same batch#

Take a batch of two prompts, four responses each, with 0/1 rewards. Prompt A is hard, $r=(1,0,0,0)$; prompt B is medium, $r=(1,1,0,0)$.

GRPO. Group A has mean 0.25 and standard deviation $\sqrt{0.25\cdot 0.75}=0.433$, so the success gets $0.75/0.433=1.732$ and each failure $-0.25/0.433=-0.577$. Group B has mean 0.5 and standard deviation 0.5, giving $\pm 1$.

REINFORCE++ with baseline. Subtract group means: A becomes $(0.75,-0.25,-0.25,-0.25)$, B becomes $(0.5,0.5,-0.5,-0.5)$. The batch mean of these is 0; the batch standard deviation is $\sqrt{(0.75+1.0)/8}=0.468$. Dividing: A's success gets 1.603, its failures $-0.534$; B's responses get $\pm 1.069$.

REINFORCE++ (batch mean only). Treat all eight rewards as one pool: mean 0.375, standard deviation 0.484. Every success gets $(1-0.375)/0.484=1.291$ and every failure $-0.774$, regardless of prompt.

With binary rewards the differences are modest: relative to GRPO, the baseline variant shifts weight from the hard prompt to the medium one by about 7%. Now make the rewards continuous, as a reward model's would be. Prompt A's four responses score $(0.71,0.70,0.70,0.69)$, differences that are within the reward model's noise; prompt B's score $(0.9,0.1,0.5,0.3)$. GRPO gives A's responses advantages $(1.394,0,0,-1.394)$, the same size as B's real differences $(1.521,-1.183,0.169,-0.507)$. REINFORCE++ with baseline gives A $(0.048,0,0,-0.048)$ and B $(2.150,-1.672,0.239,-0.717)$. The batch scale lets a prompt with nothing to learn contribute almost nothing.

Toy run: noise amplification and the KL estimators#

To test that last point beyond one batch, we built a tabular bandit: 256 prompts with 10 candidate answers each. On half the prompts every answer has true quality 0, so there is nothing to learn; on the other half qualities are drawn from $\mathcal{N}(0,1)$. Each reward is the true quality plus $\mathcal{N}(0,0.1^2)$ noise, mimicking a learned reward model. Every method gets the same 256 samples per step: GRPO and REINFORCE++ with baseline use 32 prompts with $k=8$; REINFORCE++ uses 256 prompts with $k=1$. We swept the learning rate over {3, 10, 30, 100} and ran 5 seeds of 100 steps. We report the average true quality on informative prompts (starting at 0.00; the best achievable is about 1.54) and policy entropy on the flat prompts (starting at $\ln 10=2.303$), where any loss of entropy is overfitting to noise.

At every learning rate, GRPO spends part of its update budget on noise. At a learning rate of 30, where nothing has saturated, REINFORCE++ with baseline reaches a true quality of 1.297 against GRPO's 1.105 while leaving the flat prompts essentially untouched (2.300 versus 2.234 nats). At a learning rate of 100 all three approach the ceiling on informative prompts, but GRPO has collapsed the flat prompts from 2.303 to 1.516 nats, against 2.048 for the baseline variant and 1.880 for single-sample REINFORCE++. In an LLM, those flat prompts are the ones where the reward model cannot tell responses apart, and the entropy you lose there is diversity you lose everywhere, because parameters are shared.

Two caveats. In a separate run with binary rewards and no reward noise (10 answers per prompt, one correct, 60 steps), GRPO, REINFORCE++ with baseline and RLOO all finished at the same 73.9% pass rate once the learning rate was tuned, and single-sample REINFORCE++ at 70.7%, so the advantage of batch normalisation is a robustness property, not a free accuracy gain. And the tabular model has no shared parameters, so it cannot reproduce the generalisation results (the AIME overfitting experiment) in the paper.

The training loop#

for step in range(num_steps):
    prompts = sample_batch(dataset, n_prompts)            # e.g. 128 prompts
    with no_grad():
        outs = [policy.generate(q, n=k) for q in prompts]  # k = 1 or k = 8
        r = reward(prompts, outs)                          # (n_prompts * k,)
        logp_old = token_logprobs(policy, prompts, outs)
        logp_ref = token_logprobs(ref, prompts, outs)
        if variant == "reinforce++":                       # KL folded into the reward
            kl = logp_old - logp_ref                       # k1 per token
            A = r[:, None] - beta * reverse_cumsum(kl * mask)   # return-to-go, gamma = 1
        else:                                              # "with baseline"
            A = (r - group_mean(r, k))[:, None].expand_as(mask)
        A = masked_whiten(A, mask)                         # mean/std over ALL tokens in batch
    for mb in minibatches(prompts, outs, A, logp_old, logp_ref):
        logp = token_logprobs(policy, mb)
        ratio = exp(logp - mb.logp_old)
        pg = -minimum(ratio * mb.A, clip(ratio, 1 - eps, 1 + eps) * mb.A)
        loss = masked_mean(pg, mb.mask)
        if variant != "reinforce++":
            loss += lam * masked_mean(0.5 * (logp - mb.logp_ref) ** 2, mb.mask)   # k2 loss
        loss.backward(); optimizer.step(); optimizer.zero_grad()

Two implementation details matter. First, "batch" means the global batch across all data-parallel ranks, not the per-GPU microbatch; whitening per device reintroduces small-sample statistics. Second, normalisation happens over tokens in the reference implementations (verl's masked_whiten and OpenRLHF's estimator both whiten the token-level advantage tensor under the response mask), so long responses carry more weight in the statistics than short ones. For outcome rewards without per-token KL, token-level and response-level whitening differ only through that length weighting.

Data

Prompts only, plus a reward function. For the single-sample variant, prompt diversity is the point, so use large, varied prompt sets (the paper's general-RLHF run used 20,000 prompts). For the baseline variant with verifiable rewards, the same filtering advice as GRPO applies: prompts the model always or never solves give zero group-centred advantage and are wasted, although under batch normalisation they no longer distort anyone else's scale.

Hyperparameters#

The paper's own guidance is to choose the variant by task. REINFORCE++ with one sample per prompt for general-domain RLHF, for process reward models, and wherever several rewards per prompt are expensive; it works best with symmetric rewards such as $\pm 1$. REINFORCE++ with baseline for reasoning and agent tasks, where it tolerates both 0/1 and $\pm 1$ rewards and helps when many rollouts are uninformative.

Compute and memory#

Memory is the same as GRPO: a trainable policy (about 16 bytes per parameter with Adam in mixed precision), a frozen reference for the KL term (2 bytes per parameter in bf16), a generation engine and a reward model or verifier. There is no critic, which saves roughly another 16 bytes per parameter of a same-sized value model compared with PPO:

Compute differs by variant. With $k=1$, each prompt is generated and scored once, so for the same number of responses the generation cost equals GRPO's, but coverage is $k$ times wider. Global normalisation itself requires one all-reduce of two scalars per step. Against DPO, both variants cost what online RL costs: generation dominates, and a run is one to two orders of magnitude more expensive than an offline preference pass over a similar number of examples.

Failure modes#

Weak baseline with k = 1. The batch mean ignores prompt difficulty, so on a mixture of easy and hard prompts every response to an easy prompt looks good and every response to a hard prompt looks bad. Gradient variance is higher than with a per-prompt baseline. Signs: noisy reward curve, slow progress on the hard subset. Fix: move to the baseline variant, or stratify batches by difficulty.

Asymmetric 0/1 rewards with k = 1. With 0/1 rewards and a low pass rate, the batch-centred advantage of a failure is small and negative while a rare success is large and positive; this is fine in expectation but noisy. The paper recommends $\pm 1$ for the single-sample variant.

Per-device normalisation. If your framework whitens per microbatch, you have GRPO-like small-sample statistics again. Check the code path, not the config name.

KL drift with the wrong estimator. If you port the baseline variant into a GRPO codebase, make sure the KL loss uses $k_2$, not $k_3$. A $k_3$ loss on a policy that has pushed reference-favoured tokens near zero produces huge, rare gradients; watch the maximum of ${\pi }_{\text{ref}}/{\pi }_{\theta }$ over the batch.

Reward hacking against a learned RM. The paper reports GRPO's KL rising much faster than REINFORCE++'s against a Bradley-Terry reward model, read as reward hacking. Batch normalisation slows it but does not prevent it. Track KL to the reference, response length and a held-out preference evaluation.

When to use REINFORCE++#

Minimal implementation#

import torch
import torch.distributed as dist

def global_whiten(x, mask, eps=1e-8):
    """Mean/std over all masked entries across every data-parallel rank."""
    s = torch.stack([(x * mask).sum(), (x * x * mask).sum(), mask.sum()])
    if dist.is_initialized():
        dist.all_reduce(s)
    mean = s[0] / s[2]
    var = s[1] / s[2] - mean ** 2
    return (x - mean) * torch.rsqrt(var.clamp(min=0) + eps) * mask

def rfpp_advantages(rewards, mask, k, logp_old=None, logp_ref=None, beta=0.0):
    """rewards: (N,) prompt-major; mask: (N, T). k == 1 gives REINFORCE++, k > 1 the baseline variant."""
    if k == 1:
        A = rewards[:, None].expand_as(mask).clone()
        if beta > 0:
            kl = (logp_old - logp_ref) * mask                      # k1 per token
            A = A - beta * kl.flip(1).cumsum(1).flip(1)           # sum_{i >= t} KL(i)
    else:
        centred = rewards.view(-1, k) - rewards.view(-1, k).mean(1, keepdim=True)
        A = centred.view(-1, 1).expand_as(mask).clone()
    return global_whiten(A, mask)

def k2_kl_loss(logp, logp_ref, mask):
    return (0.5 * (logp - logp_ref) ** 2 * mask).sum() / mask.sum()

Plug the advantages into any PPO-style clipped loss. In OpenRLHF the estimators are --algo.advantage.estimator reinforce and reinforce_baseline, with --algo.kl.use_loss --algo.kl.estimator k2 for the baseline variant; in verl, algorithm.adv_estimator=reinforce_plus_plus or reinforce_plus_plus_baseline. TRL approximates REINFORCE++ in RLOOTrainer via normalize_advantages=True. In POFE, REINFORCE++ with baseline is available as an advantage option on the GRPO recipe, so the two can be compared on the same prompt set and reward function.

Further reading#

Hu et al. (2025) is the primary source; read the current version, whose framing differs from the January 2025 original. Andrychowicz et al. (2020), cited for global advantage normalisation, is the large-scale study of on-policy RL implementation choices outside language models. Ahmadian et al. (2024) is the case for REINFORCE-style methods over PPO in RLHF, and Li et al. (2023) introduces ReMax. Khatri et al. (2025) and Liu et al. (2025, "Tricks or Traps") are the independent ablations that compare batch-level and group-level normalisation. Hu et al. (2024) describes OpenRLHF, where the reference implementation lives.

Frequently asked questions#

References#

1. Jian Hu, Jason Klein Liu, Haotian Xu, Wei Shen (2025). REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization (first version: REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models). arxiv.org/abs/2501.03262
2. Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, et al. (2024). OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arxiv.org/abs/2405.11143
3. Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arxiv.org/abs/2402.03300
4. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arxiv.org/abs/2402.14740
5. Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, Zhi-Quan Luo (2023). ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. arxiv.org/abs/2310.10505
6. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arxiv.org/abs/1707.06347
7. Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, et al. (2020). What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study. arxiv.org/abs/2006.05990
8. John Schulman (2020). Approximating KL Divergence. joschu.net/blog/kl-approx.html
9. Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, Rishabh Agarwal (2025). The Art of Scaling Reinforcement Learning Compute for LLMs. arxiv.org/abs/2510.13786
10. Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, et al. (2025). Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning. arxiv.org/abs/2508.08221
11. ByteDance Seed and verl contributors (2024). verl: Volcano Engine Reinforcement Learning for LLMs. github.com/volcengine/verl

Related reading