VAPO and value-based RL for long chain-of-thought, explained

VAPO (Value-model-based Augmented Proximal Policy Optimization) is a recipe for training long chain-of-thought reasoning models with PPO and a learned value function, rather than with the critic-free group baselines that became standard after DeepSeek-R1. It keeps PPO's actor-critic structure and fixes the specific ways a critic breaks on responses thousands of tokens long: a biased initial value model, a GAE decay that wipes out the reward before it reaches early tokens, and length-dependent bias and variance. This article covers VAPO together with its two closest relatives: VC-PPO, the earlier paper from the same group that introduced value pretraining and decoupled GAE, and VinePPO, which removes the value network altogether and estimates values by Monte Carlo rollouts from intermediate states. All three sit in the RL stage after SFT or a cold start, in the same slot as GRPO and DAPO, and all three argue the same thing: per-token credit assignment is worth paying for when responses are long.

Where it came from#

By early 2025, value-free methods dominated reasoning RL. PPO with a critic had a reputation for collapsing on long chain-of-thought: responses shrink, the model stops reasoning and answers directly. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan and Lin Yan at ByteDance Seed diagnosed that collapse in What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret (Yuan et al., 2025), the paper that introduced VC-PPO (Value-Calibrated PPO). Their explanation: the critic is usually initialised from the reward model, but a reward model is trained to score complete responses at the end-of-sequence token, while a value model must predict expected return at every prefix. Initialising one from the other gives a systematic bias that GAE propagates backwards through the whole response. Separately, GAE's $\lambda =0.95$ shrinks the terminal reward by $0.95^{100}\approx 0.006$ only 100 tokens back, so on long responses the critic's own targets are almost entirely bootstrapped from its own (biased) predictions.

VC-PPO's fixes were value pretraining (train the critic on Monte Carlo returns from a frozen policy before RL starts) and decoupled GAE ($\lambda =1$ for the critic's targets, $\lambda =0.95$ for the policy's advantages). On Qwen2.5-32B with a 16K context it reached 48.8 pass@1 on AIME 2024 against 38.9 for GRPO in the same setting; plain PPO in the ablation scored 5.6.

A month later, VAPO (Yue et al., 2025), from Yu Yue, Yufeng Yuan, Qiying Yu and colleagues at ByteDance Seed, combined those two fixes with five more: length-adaptive GAE, a token-level policy loss, DAPO's clip-higher, an NLL loss on correct samples, and group sampling. Starting from a Qwen 32B pre-trained model, VAPO scored 60.4 on AIME 2024, against 50 for DAPO and 47 for DeepSeek-R1-Zero-Qwen-32B, reached that within 5,000 steps, and reported no training crashes across three independent runs that each peaked at 60 to 61.

VinePPO (Kazemnejad et al., 2024), by Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville and Nicolas Le Roux at Mila, came at the problem from the other side. Measuring value networks during PPO training on MATH and GSM8K, they found them poor at ranking candidate next steps even when explained variance looked fine (0.7 to 0.9). Since a language environment can be reset to any prefix by simply feeding it back to the model, they replaced the network with $K$ sampled continuations per reasoning step, in the spirit of the "vine" variant of TRPO.

Background: what the critic is for#

Treat generation as a token-level MDP: state $s_t$ is the prompt plus tokens so far, action $a_t$ is the next token, and the only reward is $R$ at the final token. PPO maximises the clipped surrogate

written here in VAPO's final form, with the token-level normaliser $1/{\sum }_i|o_i|$ and asymmetric clip. The ratio is $r_{i,t}={\pi }_{\theta }(a_t\mid s_t)/{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)$. The advantage comes from Generalized Advantage Estimation (Schulman et al., 2015) with a value function $V$:

With $\gamma =1$ and terminal reward only, two limits are worth keeping in mind. At $\lambda =1$ the sum telescopes to ${\hat{A}}_t=R-V(s_t)$: an unbiased Monte Carlo advantage whatever the critic is, with the critic acting only as a baseline. At $\lambda =0$, ${\hat{A}}_t=V(s_{t+1})-V(s_t)$: low variance, fully dependent on the critic being right. GRPO and RLOO are, in this language, $\lambda =1$ with the critic replaced by a group-mean baseline, so every token in a response gets the same advantage. A critic is the only way, short of extra rollouts, to give different tokens of the same response different credit.

VAPO's seven modifications, with the math#

1. Value pretraining

Before policy training, sample responses from a fixed policy (the SFT model) and regress the value model on Monte Carlo returns, which with a single terminal reward and $\gamma =1$ is just the response's final reward for every prefix:

Train until value loss and explained variance stabilise, then start RL from that checkpoint. The VAPO paper's training details describe a 50-step value warm-up from the reward-model initialisation. Removing it was the single most damaging ablation: AIME 2024 dropped from 60 to 11, the same collapse seen with vanilla PPO (5).

2. Decoupled GAE

Use two different $\lambda$ values. The critic is trained towards ${\lambda }_{\text{critic}}=1$ targets, ${\hat{R}}_t=R$, which are unbiased and do not decay with distance from the end. The policy uses advantages from a smaller ${\lambda }_{\text{policy}}$, trading some bias for lower variance and faster convergence. Removing decoupling cost 27 points (60 to 33).

3. Length-adaptive GAE

A fixed ${\lambda }_{\text{policy}}$ treats a 200-token and an 8,000-token response very differently. VAPO sets the effective horizon of the GAE sum proportional to the response length $l$:

This gives $\lambda =0.9$ at 200 tokens, 0.98 at 1,000 and 0.9975 at 8,000. A useful way to read it: the terminal reward's weight on a token at fractional distance $x$ from the end is ${\lambda }^{xl}\approx e^{-x/\alpha }=e^{-20x}$, the same curve for every length (Figure 2). One consequence the paper does not discuss: the formula is only meaningful for $l>1/\alpha =20$ tokens. Below that $\lambda$ is zero or negative, so implementations need a floor. Removing length adaptation cost 15 points.

4. Token-level policy loss

Average the per-token PPO terms over all tokens in the batch rather than first within each response (the normaliser in the loss above). Long responses then carry weight proportional to their length, which matters because problems in long responses (repetition, drift) are exactly what needs suppressing. Taken from DAPO; worth 7 points.

5. Clip-higher

Separate clip bounds, ${\epsilon }_{\text{low}}=0.2$ and ${\epsilon }_{\text{high}}=0.28$, leave more room to raise low-probability tokens without letting the model push tokens to zero, which counters entropy collapse. Also from DAPO; removing it limited the run to 46.

6. Positive-example LM loss

When most samples are wrong, the rare correct ones are expensive. VAPO adds a negative log-likelihood term on correct samples $\mathcal{T}$:

This is rejection-sampling fine-tuning run inside the RL loop. Worth about 6 points.

7. Group sampling

For a fixed budget, sample fewer prompts more times: VAPO uses 512 prompts with 16 samples each instead of the baseline's 8,192 prompts with one sample. Contrasting correct and incorrect responses to the same prompt gives a richer signal even with a critic; worth 5 points.

VinePPO: values without a value network#

VinePPO keeps PPO but replaces $V_{\varphi }(s_t)$ with an empirical mean. From an intermediate state $s_t$ of a training trajectory, sample $K$ continuations ${\eta }_1,\dots ,{\eta }_K\sim {\pi }_{\theta }(\cdot \mid s_t)$ and average their returns:

The auxiliary rollouts are used only for value estimation, never as training samples. For any $K\ge 1$ the resulting policy gradient is unbiased; $K$ only controls variance. To keep cost manageable, VinePPO estimates values at reasoning-step boundaries (lines or sentences) and assigns each step's advantage to all its tokens. Default $K=9$. With DeepSeekMath 7B and RhoMath 1.1B on MATH and GSM8K it beat PPO, GRPO, RLOO, RestEM and DPO+, and it reached PPO's peak accuracy in 2.8 and 9 times fewer steps and 1.51 and 3.0 times less wall-clock time (7B and 1.1B respectively), even though each iteration was up to 2 and 5 times slower.

Put the three approaches side by side, for a token in a response with terminal reward $R$:

Worked example: GAE on five tokens#

A five-token response earns $R=1$. The critic predicts $V=(0.50,0.55,0.40,0.70,0.80)$ at the five prefixes, and $V(s_5)=0$ at the terminal state. With $\gamma =1$, the TD errors are

At $\lambda =1$ the advantages are $R-V(s_t)=(0.50,0.45,0.60,0.30,0.20)$. Every token is credited positively; the ordering follows the critic's baseline, not what each token did. At $\lambda =0.5$, working backwards with ${\hat{A}}_t={\delta }_t+0.5{\hat{A}}_{t+1}$:

Now token 2, the one after which the critic's estimate jumped from 0.40 to 0.70, gets the most credit, and token 1, after which the estimate fell, gets almost none. That is the per-token credit a critic buys, and it is only as good as the critic: if $V$ is biased, those differences are wrong. At $\lambda =0.95$ the numbers are $(0.427,0.397,0.576,0.290,0.200)$, close to the $\lambda =1$ values because five tokens is short. The decoupling in VC-PPO and VAPO is exactly this split: train the critic towards the honest $\lambda =1$ targets, and use a smaller $\lambda$ only for the policy's advantages.

Toy run: which estimator finds the mistake?#

Credit assignment is easy to state and hard to see, so we simulated it directly. A response is 1,000 tokens split into 20 reasoning steps of 50 tokens. Each token has a 0.07% chance of being a fatal error, after which the answer is wrong; otherwise the answer is right. That gives a 49.3% success rate over 2,000 simulated rollouts, 1,013 of them failures. The true value of a prefix is known in closed form, $(1-0.0007{)}^{T-t}$ before the error and 0 after it. For each failed rollout we compute per-token advantages with several estimators, average them within each step, and ask whether the step containing the fatal token gets the most negative advantage. Chance is 5%.

The estimators: GAE at $\lambda =0.95$ with the exact value function; the same with a noisy critic (smooth AR(1) error, standard deviation 0.15, correlation 0.99 per token); the noisy critic with an extra constant bias of +0.3; GAE at $\lambda =1$ with the noisy critic; and VinePPO-style Monte Carlo values at step boundaries with $K=1$, 3 and 9.

Three things stand out. First, $\lambda =1$ finds the error only 8.8% of the time even though its policy gradient is unbiased: $R-V(s_t)$ mostly reflects the critic's baseline, not which step mattered. This is why VAPO keeps ${\lambda }_{\text{policy}}<1$ and why a value model is needed at all. Second, GAE at 0.95 does well with a good critic (78.5% exact, 75.6% noisy) and loses about 9 points when the critic is biased (66.8%), the mechanism VC-PPO's value pretraining targets. Even the exact critic misses about a fifth of the time, because GAE smears credit roughly 20 tokens backwards, into the previous step when the error falls early in a step. Third, Monte Carlo values are poor with one rollout (35.0%), competitive with three (66.5%) and best with nine (93.9%), which is VinePPO's default for a reason. The toy has a single fatal-error mechanism and a known value function, so it flatters any method that estimates values well; it says nothing about how hard a real 32B value model is to train.

The VAPO training loop#

# Phase 0: value pretraining (frozen policy, Monte Carlo targets)
for step in range(value_warmup_steps):                       # VAPO: about 50 steps
    outs = policy_sft.generate(sample(prompts), n=1)
    R = verifier(outs)                                       # +1 / -1 or 1 / 0
    value_loss = mse(value(outs), R.broadcast_to_tokens())   # lambda = 1 targets
    value_loss.backward(); value_opt.step()

# Phase 1: RL
for step in range(num_steps):
    batch = sample(prompts, 512)
    outs = policy.generate(batch, n=16, temperature=1.0)     # group sampling
    R = verifier(outs)
    with no_grad():
        V = value(outs)                                      # (N, T+1), V[:, T] = 0
        logp_old = policy.logprobs(outs)
        lam_pol = clamp(1 - 1 / (0.05 * outs.lengths), min=0.0)  # length-adaptive, per response
        A = gae(R, V, gamma=1.0, lam=lam_pol)                # policy advantages
        returns = gae(R, V, gamma=1.0, lam=1.0) + V[:, :-1]  # critic targets (= R)
    for mb in minibatches(outs, A, returns, logp_old, size=512):
        ratio = exp(policy.logprobs(mb) - mb.logp_old)
        pg = -minimum(ratio * mb.A, clip(ratio, 1 - 0.2, 1 + 0.28) * mb.A)
        loss = token_mean(pg, mb.mask)                       # over all tokens in the minibatch
        loss += 0.1 * token_mean(-policy.logprobs(mb), mb.mask * mb.correct[:, None])
        loss.backward(); policy_opt.step()
        value_loss = token_mean((value(mb) - mb.returns) ** 2, mb.mask)
        value_loss.backward(); value_opt.step()

Data is the same as for any reasoning RL run: prompts with verifiable answers. VC-PPO trained on past AIME problems plus constructed hard problems, with a cold start of "dozens" of formatted examples to teach the <think> format; rewards were +1 for correct and $-1$ for incorrect, with KL coefficient 0 because a rule-based reward is hard to hack. For VinePPO, the extra step is splitting responses into reasoning steps (newlines work for maths solutions) and generating $K$ continuations from each step boundary.

Hyperparameters#

The VC-PPO paper also swept ${\lambda }_{\text{actor}}$ between 0.9 and 1.0 with ${\lambda }_{\text{critic}}$ fixed at 1, and VAPO's evaluation used avg@32 on AIME 2024 with top-p 0.7 and temperature 1.0, which is worth matching when comparing against its numbers.

Compute and memory#

VAPO pays PPO's full memory bill. With full fine-tuning in mixed precision and Adam, each trainable model costs about 16 bytes per parameter, and the critic is usually the same size as the policy:

For a 32B policy and a 32B critic that is roughly 1 TB of training state before activations, so the critic alone adds about 512 GB, the price of the per-token signal. Per step, the critic adds a forward pass over every rollout and a forward and backward pass for its update; value pretraining is a one-off phase of generation plus critic training. A smaller critic cuts this but makes the bias problem harder.

VinePPO has no critic memory but multiplies generation. With $S$ step boundaries per response and $K$ rollouts from each, the extra tokens generated per response are roughly

so for $K=9$, $S=20$ and $l=1,000$, about 90 times the response itself, before any prefix caching. VinePPO's measured slowdown per iteration was 2 to 5 times on MATH-scale solutions, which shows how much an inference engine with prefix caching absorbs; but the formula grows with $S\cdot l$, and for responses of tens of thousands of tokens it is the reason long-CoT recipes such as VAPO kept a learned critic.

Failure modes#

Length collapse. The VC-PPO signature: mean response length falling steadily early in training while accuracy stalls and the model starts answering without reasoning. Cause: a biased critic. Check explained variance of the critic on fresh rollouts before and during training; if it is low or negative at step 0, you skipped or under-ran value pretraining.

Critic lag. Value loss rising as the policy improves means the critic cannot track the policy. Raise the critic learning rate relative to the actor (VAPO uses 2 times) or take more critic steps per batch.

Entropy too low. VAPO's entropy ended lower than DAPO's; the authors judged the stability worth it. If pass@k on held-out problems starts falling while pass@1 rises, exploration is shrinking. Raise ${\epsilon }_{\text{high}}$ or lower $\mu$.

NLL overfitting. With few prompts and a large $\mu$, the model memorises its own correct solutions. Watch the gap between training and held-out accuracy.

Short responses with adaptive $\lambda$. For responses under 20 tokens, ${\lambda }_{\text{policy}}$ is negative unless clamped; for responses just above, it is near 0 and the advantage is pure one-step critic difference. Mixed-format tasks with very short answers need a floor on $\lambda$.

VinePPO variance at low $K$. With $K=1$ the advantage at each step is the difference of two Bernoulli draws, mostly noise; our toy identified the right step 35% of the time. If you cannot afford $K\ge 3$, use fewer, longer steps rather than fewer rollouts.

When to use VAPO, VinePPO or a value-free method#

Minimal implementation#

The parts that differ from a standard PPO trainer are the GAE with a per-response $\lambda$, the decoupled targets and the extra NLL term. A vectorised sketch in PyTorch:

import torch

def gae_per_seq(rewards, values, mask, lam):
    """rewards, mask: (N, T); values: (N, T+1) with values[:, T] = 0; lam: (N,) or float."""
    N, T = rewards.shape
    lam = torch.as_tensor(lam, dtype=values.dtype, device=values.device).expand(N)
    adv = torch.zeros_like(rewards); last = torch.zeros(N, device=rewards.device)
    for t in reversed(range(T)):
        delta = rewards[:, t] + values[:, t + 1] * mask[:, t] - values[:, t]
        last = (delta + lam * last) * mask[:, t]
        adv[:, t] = last
    return adv

def vapo_targets(rewards, values, mask, alpha=0.05, lam_floor=0.0):
    lengths = mask.sum(1).clamp(min=1)
    lam_policy = (1 - 1 / (alpha * lengths)).clamp(min=lam_floor)   # length-adaptive
    adv = gae_per_seq(rewards, values, mask, lam_policy)
    returns = gae_per_seq(rewards, values, mask, 1.0) + values[:, :-1]  # lambda_critic = 1
    return adv, returns

def vapo_policy_loss(logp, logp_old, adv, mask, correct, eps_low=0.2, eps_high=0.28, mu=0.1):
    ratio = torch.exp(logp - logp_old)
    pg = -torch.minimum(ratio * adv, ratio.clamp(1 - eps_low, 1 + eps_high) * adv)
    pg_loss = (pg * mask).sum() / mask.sum()                         # token-level
    pos = mask * correct[:, None].float()
    nll = -(logp * pos).sum() / pos.sum().clamp(min=1)
    return pg_loss + mu * nll

def vine_values(policy, prefixes, verifier, K=9):
    """Monte Carlo value at each step boundary: mean reward of K continuations."""
    outs = policy.generate(prefixes, n=K)                            # (len(prefixes) * K)
    return verifier(outs).view(-1, K).float().mean(1)

verl, the framework VC-PPO was trained with, ships GAE-based PPO with separate actor and critic configurations; the length-adaptive $\lambda$, decoupled targets and NLL term are a few lines on top. The VinePPO authors released their code at the repository linked from the paper. In POFE, value-based PPO is a recipe option with a value-pretraining phase and critic explained variance shown on the run page, because that one number predicts most PPO collapses on long responses.

Further reading#

Yue et al. (2025) for VAPO and its ablation table; Yuan et al. (2025) for the VC-PPO diagnosis of value-initialisation bias and the first version of decoupled GAE; Kazemnejad et al. (2024) for VinePPO and the most careful measurement of how wrong value networks are on reasoning. Schulman et al. (2015) defines GAE and Schulman et al. (2017) PPO. Yu et al. (2025) for DAPO, from which VAPO takes clip-higher and the token-level loss, and Hu et al. (2025) for Open-Reasoner-Zero, a value-based PPO recipe that uses Monte Carlo ($\lambda =1$) estimates instead of GAE.

Frequently asked questions#

References#

1. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, et al. (2025). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arxiv.org/abs/2504.05118
2. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, Lin Yan (2025). What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret. arxiv.org/abs/2503.01491
3. Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, Nicolas Le Roux (2024). VinePPO: Refining Credit Assignment in RL Training of LLMs. arxiv.org/abs/2410.01679
4. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. arxiv.org/abs/1506.02438
5. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arxiv.org/abs/1707.06347
6. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arxiv.org/abs/2503.14476
7. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, Heung-Yeung Shum (2025). Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model. arxiv.org/abs/2503.24290
8. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948
9. McGill-NLP (2024). VinePPO code repository. github.com/McGill-NLP/VinePPO

Related reading