RLVR explained: reinforcement learning with verifiable rewards

Reinforcement learning with verifiable rewards (RLVR) is post-training in which the reward is computed by a program that checks the model's output against ground truth, instead of by a reward model trained on human preferences. A maths answer is compared with the reference after normalisation, generated code is run against unit tests, a JSON extraction is validated against a schema and expected values, an instruction like "answer in exactly three bullet points" is checked by a parser. The policy is then optimised with an on-policy RL algorithm, most often GRPO or one of its descendants, sometimes PPO. RLVR is a training setup more than a single algorithm: what defines it is where the reward comes from. In the post-training stack it usually follows SFT and preference tuning, or, in "zero" recipes, replaces them and runs directly on a base model.

Where the term and the method come from#

Checking answers programmatically is old. Cobbe et al. (2021) released GSM8K with the explicit aim of training verifiers for maths word problems, and Uesato et al. (2022) compared outcome-based and process-based feedback on the same kind of problems. The name RLVR was introduced by Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison and colleagues at the Allen Institute for AI in the Tülu 3 report (Lambert et al., 2024), which defines it as "an RL-based method that only gets a reward if the model's completions are verified to be correct." Tülu 3 applied it as a final stage after DPO, using PPO with a reward of $\alpha =10$ for a verified answer and 0 otherwise, on 29,946 prompts: 7,473 from GSM8K, 7,500 from MATH and 14,973 instruction-following prompts with constraint-specific verifiers.

Two months later DeepSeek-R1 (DeepSeek-AI, 2025) showed what the setup can do at scale: GRPO with rule-based accuracy and format rewards, applied directly to a base model, produced long chain-of-thought reasoning competitive with the strongest closed models on maths and code. That result made RLVR the default way to train reasoning models and set off a year of follow-up work on algorithms (Dr. GRPO, DAPO) and on what RLVR actually changes inside the model.

Why verifiable rewards change the picture#

Classic RLHF trains a reward model $r_{\varphi }$ on human comparisons and optimises the policy against it. A learned reward is a proxy; push hard enough against it and the policy finds inputs where the proxy is wrong. Gao et al. (2022) measured this over-optimisation as a scaling law: true quality first rises and then falls as KL from the initial policy grows. The KL penalty in RLHF exists largely to stop that.

A verifier is not a proxy in the same sense. If the reference answer is right and the comparison is exact, a reward of 1 means the answer is correct, full stop. There is no smooth surface to exploit, so the policy can be allowed to move much further from its initialisation, and many RLVR recipes drop the KL term entirely. The price is coverage: verifiers only exist for tasks with checkable outcomes, and they score only the final answer, not whether the reasoning that led to it was sound.

The objective#

Let $x$ be a prompt from a dataset $\mathcal{D}$, $y\sim {\pi }_{\theta }(\cdot \mid x)$ a sampled completion, and $v(x,y)\in \{0,1\}$ a deterministic verifier that returns 1 when the answer extracted from $y$ matches the ground truth for $x$. RLVR maximises

where $\alpha$ scales the correctness reward (10 in Tülu 3, 1 in most GRPO recipes, or $\pm 1$ as in DAPO), $f$ is an optional format term (for example, 1 if the reasoning sits inside the expected tags) with weight $\lambda$, ${\pi }_{\text{ref}}$ is the frozen starting policy, and $\beta \ge 0$ sets the KL penalty (Tülu 3 swept values from 0.001 to 0.1; reasoning recipes often use 0).

The gradient is the ordinary policy gradient. With a baseline $b(x)$, which GRPO estimates from a group of samples and PPO from a critic,

For a binary verifier and a prompt on which the policy currently succeeds with probability $p(x)$, the expected reward is simply $\alpha p(x)$. RLVR is therefore direct optimisation of pass@1 on the training prompts. Everything interesting about it comes from how that interacts with the rest of the distribution: which other prompts improve, which behaviours get amplified, and what happens to the probability of answers the model rarely produced.

RLVR next to its neighbours

Rejection sampling with a verifier is the closest relative. It optimises the same objective with a cruder estimator: it uses only the positive samples, gives every one equal weight, and ignores failures. RLVR's advantage-weighted update also pushes down what failed, which is why it typically extracts more from the same rollouts.

Verifiers in practice#

Almost every RLVR failure traces back to the verifier, so it deserves more engineering attention than the RL algorithm. A verifier has three parts: an extractor that finds the answer in free text, a normaliser that maps equivalent forms to one representation, and a comparator or executor.

The test for a verifier is not whether it accepts correct answers. It is whether it rejects everything else. Before training, sample several hundred completions from the starting policy, have a person label them, and measure the verifier's false-positive and false-negative rates. A false-negative rate of a few percent mostly adds noise. A false-positive rate of a few percent on a behaviour the policy can learn becomes, under optimisation, the main thing the policy learns. The toy below shows how fast.

Worked example: scoring one group#

A prompt asks for the probability of rolling a sum of 7 with two dice; the reference answer is $1/6$. Eight sampled completions end with: "1/6", "0.1667", "$\frac{1}{6}$", "6/36", "1/12", "the answer is 1/6 or 1/12", "7/36" and a response truncated before any answer.

A naive exact-string verifier gives $r=(1,0,0,0,0,0,0,0)$. After normalisation to a rational (with a tolerance for decimals and a rule that rejects responses containing more than one candidate), it gives $r=(1,1,1,1,0,0,0,0)$. These are very different training signals. Under GRPO with population std, the naive verifier gives the one exact match an advantage of $(1-0.125)/0.331=2.65$ and every other completion $-0.378$, so three correct answers written differently are actively pushed down. The normalised verifier gives $\pm 1.0$: four correct answers up, four wrong ones down. Had the verifier accepted any response containing "1/6" anywhere, the hedged "1/6 or 1/12" would have scored 1, and listing several answers would start being reinforced on every prompt the model is unsure about.

With Tülu 3's reward scale ($\alpha =10$) and PPO, the same group gives raw rewards of 10 or 0, and the KL penalty enters per token as $\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$. At $\beta =0.05$, a 300-token response whose tokens average 0.1 nats of divergence pays $0.05\times 300\times 0.1=1.5$ reward units, against 10 for being right. The ratio $\alpha /\beta$ is what actually controls how far the policy will drift for a verified answer.

Two toy runs: what RLVR amplifies#

Sharper pass@1, narrower pass@k

Yue et al. (2025) measured pass@k for RLVR-trained models and their base models at large $k$ and found that the RL models win at small $k$ but the base models catch up and overtake them as $k$ grows: RLVR mostly re-weights solutions the base model could already sample, and loses some it could. Our first toy reproduces the mechanism with shared parameters. A policy chooses one of 12 solution strategies with a single shared softmax. Strategy 0 is broadly useful (it succeeds with probability 0.3 to 0.8 on about 60% of 400 problems); the other 11 are niche (probability 0.4 to 0.9 on a random 12% of problems each, 0.02 elsewhere). The policy starts uniform and is trained with GRPO ($G=8$, 64 problems per step, 300 steps, 5 seeds). Every number below is the script's output.

RL put 97.4% of the probability on the broadly useful strategy (entropy fell from 2.48 to 0.18 nats). That is the right move for pass@1. It is the wrong move for problems that only a niche strategy solves, and those are exactly the ones the base model could reach by sampling many times. Whether this matters depends on how you deploy: for single-shot answers RLVR is a clear win; for best-of-n sampling with a verifier at inference time, a less sharpened model may be better. Measure pass@k at the $k$ you will actually use.

A leaky verifier

The second toy has 256 prompts, each a softmax over ten outputs: one correct, eight wrong, and a "hedge" output (think: listing several answers). The correct answer's logit is per-prompt; the hedge's logit is shared across prompts, because a behaviour like hedging generalises. A leaky verifier accepts the hedge with probability $q$. GRPO, $G=8$, 32 prompts per step, 200 steps, 5 seeds.

A 30% leak on one behaviour cut final true accuracy from 68.5% to 24.4% while the training reward kept rising. A 60% leak drove true accuracy to zero, because for most prompts a hedge that passes 60% of the time beats a correct answer the model finds 7% of the time. The reward curve alone never shows the problem. This is reward hacking with a verifier, and it is the most common way RLVR runs fail in practice.

The training loop and the data#

dataset = [{"prompt": p, "truth": t} for p, t in load_verifiable_prompts()]
dataset = filter_by_pass_rate(dataset, policy, n=16, lo=0.05, hi=0.95)   # drop always/never solved

for step in range(num_steps):
    batch = sample(dataset, B)
    outs = [policy.generate(ex["prompt"], n=G) for ex in batch]
    R = [[alpha * verify(o, ex["truth"]) + lam * format_ok(o) for o in g] for g, ex in zip(outs, batch)]
    update = grpo_step   # or ppo_step / rloo_step / dapo_step
    update(policy, batch, outs, R, beta=beta)
    if step % eval_every == 0:
        log(pass_at_1(policy, heldout), strict_grader_accuracy(policy, audit_set),
            mean_len(outs), zero_variance_rate(R), entropy(policy, outs))

Data format. Each record needs a prompt and whatever the verifier needs: a reference answer, a test file, a fixture database, a constraint spec. Keep the ground truth out of the prompt. Size. Tülu 3 used about 30K prompts; DAPO-Math-17K has 17K; Wang et al. (2025) showed that on Qwen2.5-Math-1.5B even a single well-chosen example raised MATH500 from 36.0% to 73.6%, matching a 1.2K-example subset. Narrow domain tasks commonly work with 1K to 10K prompts. Difficulty. More important than count. Sample 8 to 16 completions per prompt from the starting policy and keep prompts whose pass rate is neither 0 nor 1; prompts outside that band give zero advantage under group baselines.

Hyperparameters that matter#

Compute and memory#

Memory depends on the algorithm, not on RLVR itself, but RLVR is cheaper than RLHF because there is no reward model on the GPU (the verifier is a CPU process) and often no reference model. With GRPO and $\beta =0$, full fine-tuning needs about 16 bytes per parameter of training state plus an inference copy and its KV cache; LoRA brings training state down to roughly the frozen bf16 weights plus the adapter. Tülu 3's PPO-based RLVR keeps a value model as well, roughly doubling training state.

Time is dominated by generation, as in all on-policy RL. A rule of thumb: one RLVR step costs $G$ times the decoding of an inference batch of the same prompts, plus roughly three forward-equivalents per generated token for training. Code verifiers add sandbox execution time, which can rival generation for test-heavy tasks; run them asynchronously and cap their runtime. For reference points, Tülu 3 reports its final 405B RLVR run took 46 hours on 256 GPUs, and Liu et al. (2025) trained a 7B model to strong maths results in 27 hours on 8 A100s.

Failure modes and how to detect them#

Verifier hacking. Training reward rises, held-out accuracy under a strict grader does not. Keep an audit set graded by a stricter checker or by people, track it on the same plot as training reward, and read samples of positively rewarded outputs every few hundred steps.

Format collapse or format-only gains. On base models, much of the early gain is the model learning the answer format the verifier expects. Wang et al. (2025) separate "format correction" from real gains for exactly this reason. Report accuracy with a lenient extractor as well as the training one, so format learning is not mistaken for reasoning.

Diversity loss. pass@1 rises while pass@k at large $k$ falls, entropy drops steadily, and groups become identical. Watch entropy and the zero-variance rate; consider clip-higher, a higher temperature, or a small entropy bonus.

Length runaway. Mean response length climbing without accuracy gains, especially on incorrect answers. Use a token-level or constant normaliser rather than per-sequence averaging, and plot length by correctness.

Off-target regression. RLVR on maths alone can degrade chat quality, safety behaviour or instruction following. Keep a broad regression suite and, if needed, mix in other verifiable tasks or a light KL anchor.

Contamination. If training prompts overlap the evaluation set, verifiable rewards will happily memorise answers. Decontaminate before training, as Tülu 3 did for its prompt sets.

When to use RLVR#

Minimal implementation#

The verifier is where the code effort goes. A maths verifier with normalisation, and a hookup to an open-source GRPO trainer:

import re
from fractions import Fraction

def boxed_answers(text: str):
    """Return the contents of every \\boxed{...}, respecting nested braces."""
    out, i = [], text.find("\\boxed{")
    while i != -1:
        j, depth = i + len("\\boxed{"), 1
        while j < len(text) and depth:
            depth += {"{": 1, "}": -1}.get(text[j], 0)
            j += 1
        if depth == 0:
            out.append(text[i + len("\\boxed{"): j - 1])
        i = text.find("\\boxed{", j)
    return out

def to_number(s: str):
    s = s.strip().replace(" ", "").replace("\\dfrac", "\\frac")
    m = re.fullmatch(r"\\frac\{(-?\d+)\}\{(\d+)\}", s)
    if m:
        return Fraction(int(m.group(1)), int(m.group(2)))
    try:
        return Fraction(s).limit_denominator(10_000)     # handles "1/6", "0.1667", "6/36"
    except (ValueError, ZeroDivisionError):
        return None

def verify_math(completion: str, truth: str, tol=1e-3) -> float:
    found = boxed_answers(completion)
    if len(found) != 1:                                  # none, or hedged with several answers
        return 0.0
    a, b = to_number(found[0]), to_number(truth)
    if a is None or b is None:
        return float(found[0].strip() == truth.strip())
    return float(abs(float(a) - float(b)) <= tol * max(1.0, abs(float(b))))

# Hook into TRL's GRPOTrainer: reward functions receive completions plus dataset columns.
def accuracy_reward(completions, truth, **kw):
    return [verify_math(c, t) for c, t in zip(completions, truth)]

# from trl import GRPOConfig, GRPOTrainer
# GRPOTrainer(model=ckpt, reward_funcs=[accuracy_reward],
#             args=GRPOConfig(num_generations=8, learning_rate=1e-6, beta=0.0),
#             train_dataset=ds).train()

Run on the eight completions from the worked example (with the answers wrapped in \boxed{}), this verifier returns exactly $(1,1,1,1,0,0,0,0)$: the hedged answer is rejected because it contains two boxed answers. For code, replace verify_math with a sandboxed test runner (separate process, no network, wall-clock and memory limits, and tests the model cannot read). In POFE, an RLVR recipe starts from the verifier rather than the algorithm: measure its false-positive rate on a labelled sample before training, and keep the strict-grader audit next to the training reward throughout.

Further reading#

Lambert et al. (2024) for the definition and a careful PPO-based recipe with ablations; DeepSeek-AI (2025) for RLVR at scale on a base model; Yue et al. (2025), Shao et al. (2025) and Wang et al. (2025) for what RLVR does and does not change; Gao et al. (2022) for why learned rewards needed the KL leash that verifiable rewards can sometimes drop.

Frequently asked questions#

References#

1. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arxiv.org/abs/2411.15124
2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, et al. (2021). Training Verifiers to Solve Math Word Problems. arxiv.org/abs/2110.14168
4. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins (2022). Solving math word problems with process- and outcome-based feedback. arxiv.org/abs/2211.14275
5. Leo Gao, John Schulman, Jacob Hilton (2022). Scaling Laws for Reward Model Overoptimization. arxiv.org/abs/2210.10760
6. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. arxiv.org/abs/2504.13837
7. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, et al. (2025). Reinforcement Learning for Reasoning in Large Language Models with One Training Example. arxiv.org/abs/2504.20571
8. Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, et al. (2025). Spurious Rewards: Rethinking Training Signals in RLVR. arxiv.org/abs/2506.10947
9. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arxiv.org/abs/2503.20783

Related reading