Process reward models

A process reward model (PRM) scores each step of a multi-step solution rather than only the final answer. Given a math problem and a chain of reasoning split into steps, it outputs, for every step, an estimate of whether that step is correct (or how much it moved the solution toward a correct answer). Its counterpart, the outcome reward model (ORM), gives one score per complete response, the way a standard Bradley-Terry reward model or a final-answer checker does. In the post-training stack PRMs serve three roles: verifiers that rerank many sampled solutions at inference time, guides for step-level search, and dense reward signals for RL methods such as PPO or GRPO on reasoning tasks. They are the main tool for the credit-assignment problem that outcome rewards leave open: when a 40-step solution is wrong, which step was the mistake?

The problem and where it came from#

Outcome supervision has two weaknesses on long reasoning. It is sparse: one bit for an entire trace gives the learner little information about where things went wrong. And it rewards the wrong thing sometimes: a solution with a flawed derivation can land on the right number by luck or by compensating errors, and an outcome reward pays it in full.

Uesato et al. (DeepMind, 2022) ran the first careful comparison on GSM8K grade-school math. Outcome-based supervision reached similar final-answer error rates with less labelling, but producing correct reasoning required process-based supervision, or a learned reward model that emulated it. Their best system cut final-answer error from 16.8% to 12.7% and, among solutions with correct final answers, cut reasoning error from 14.0% to 3.4%. Lightman et al. (OpenAI, 2023), Let's Verify Step by Step, scaled the comparison to the harder MATH dataset with human step-level labels. Process supervision significantly outperformed outcome supervision for training verifiers: their PRM, used to pick the best of many samples, solved 78% of problems from a representative subset of the MATH test set. They released PRM800K, the 800,000 step-level human labels behind it, and showed that active learning (labelling the solutions the current PRM finds most convincing but which reach wrong answers) made labelling much more efficient.

Human step labels are expensive, so the next wave automated them. Math-Shepherd (Wang et al., 2023) labelled each step by rolling out completions from it and checking final answers, and used the resulting PRM both for reranking and for step-by-step PPO, raising Mistral-7B from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH. OmegaPRM (Luo et al., 2024) made the rollouts cheaper with a tree search that binary-searches for the first error, collecting over 1.5 million step annotations without humans. Setlur et al. (2024) argued that a step's reward should measure progress, the change in the probability of eventually succeeding, and that this should be measured under a separate "prover" policy. On the other side of the ledger, Zhang et al. (2025) found Monte Carlo labels noticeably worse than human or LLM-judge labels, ProcessBench (Zheng et al., 2024) showed many open PRMs fail to locate errors on competition-level problems, and the DeepSeek-R1 report listed PRMs among its unsuccessful attempts, citing reward hacking and the difficulty of defining and labelling steps at scale.

The objective#

Setup

A solution to problem $x$ is a sequence of steps $s_1,\dots ,s_K$, usually split on newlines or an explicit step marker. A PRM $f_{\varphi }$ reads the prefix and outputs a step score

implemented as a sigmoid (or two-way softmax) on the hidden state at the last token of step $k$. One forward pass over the full solution produces all $K$ scores, because a causal transformer's state at the end of step $k$ sees only $s_{1:k}$.

Training loss

With a step label $z_k\in \{0,1\}$ (or a soft label in $[0,1]$), the PRM is trained with per-step binary cross-entropy:

In PRM800K-style data labels stop at the first incorrect step, since later steps built on a wrong premise have no clean label; the loss then runs over $k\le k^{\ast }$, the first error. Compare the ORM loss, which is the same cross-entropy applied once, at the last token, with $z=1[\text{final answer correct}]$, or the Bradley-Terry loss on pairs of complete responses.

Where step labels come from

Human labels. Annotators mark each step positive, negative or neutral. Most accurate, most expensive.

Monte Carlo (MC) estimation. From each prefix $s_{1:k}$, sample $N$ completions with a completer policy $\mu$ and check final answers. The empirical success rate estimates the prefix value

which approximates $V^{\mu }(x,s_{1:k})$, the probability of eventually succeeding from that prefix. Math-Shepherd's "hard" label is $z_k=1[{\hat{v}}_k>0]$ (any completion succeeds) and its "soft" label is $z_k={\hat{v}}_k$.

Progress (advantage) labels. Setlur et al. define the process reward of step $k$ as the change in value it causes,

the step-level advantage from RL. A step that keeps success probability unchanged earns roughly zero; a step that wrecks the solution earns a large negative value; a step that makes a hard problem tractable earns a large positive one. Absolute values $V$ conflate "this step was good" with "this problem is easy"; progress separates them.

LLM-judge labels. A strong model is prompted to critique each step. Zhang et al. found combining it with MC estimation and keeping only steps where the two agree (consensus filtering) gave better PRMs with less data.

From step scores to a solution score

For reranking you need one number per solution. Common choices:

Lightman et al. used the product, interpreting it as the probability that every step is correct. The product falls with length even when every step is good, so it favours short solutions; the minimum tracks the weakest step and is length-neutral. Our toy run below measures both effects.

PRMs as dense RL rewards

In step-level PPO the PRM's score is placed at the last token of each step, $r_{t_k}=g(p_k)$, and GAE spreads it over that step's tokens, so the policy gets feedback near the mistake instead of only at the end. With a progress-style reward, $r_{t_k}=A_k$, the step rewards telescope: their sum is $V(s_{1:K})-V(\varnothing )$, close to the outcome reward minus a baseline, so dense rewards change credit assignment without changing what is ultimately optimised. With raw scores $p_k$ there is no such guarantee, and the policy can learn to produce many short, safe-looking steps that each score well.

Relation to its siblings

Worked example by hand#

A four-step solution gets PRM step scores 0.95, 0.90, 0.30 and 0.85. Step 3 is the suspect. The four aggregations disagree sharply: the minimum is 0.30, the product $0.95\times 0.90\times 0.30\times 0.85=0.218$, the mean 0.75, and the last-step score 0.85. An ORM that only sees the final step and a plausible-looking answer might also say something near 0.85. If this solution's final answer happens to be right, that is the "correct answer, flawed process" case: an ORM and the last-step aggregate both accept it, while min and product flag it.

Now label the same steps by Monte Carlo with $N=8$ completions each. From the prefix ending at step 1, 7 of 8 completions reach the right answer (${\hat{v}}_1=0.875$); after step 2, 5 of 8 (${\hat{v}}_2=0.625$); after step 3, only 1 of 8 (${\hat{v}}_3=0.125$). The hard label marks every step "correct" because each prefix had at least one success, including step 3. The progress of step 3, $0.125-0.625=-0.50$, is the signal that locates the error. That single lucky completion after step 3 is exactly the failure mode the toy run below quantifies.

Building and using a PRM step by step#

# 1. data
for problem x in training problems (with reference answers):
    sample M solutions from the policy, split into steps
    for each solution, label steps:
        human:  first-error annotation
        or MC:  for k in 1..K: roll out N completions from s_1:k, v_k = success rate
                z_k = 1[v_k > 0] (hard) | v_k (soft) | v_k - v_(k-1) (progress)
        optional: keep only steps where MC and an LLM judge agree
# 2. training
init from a strong model in the same domain (often the policy's base or SFT model)
insert a step marker token after each step; read the PRM head at each marker
loss = BCE over steps up to the first error; 1 to 2 epochs
# 3. use
best-of-N:  score N solutions, aggregate with min (or product), pick the max
            or weighted majority vote: sum aggregated scores per distinct final answer
search:     beam search over steps, keeping the top-b prefixes by PRM score
RL:         put g(p_k) or the progress A_k at each step boundary as a dense reward

Data sizes. PRM800K is 800k human step labels on model-generated solutions to MATH problems. Automated datasets run from hundreds of thousands to a few million steps. MC labelling cost is $M\times K\times N$ completions: 10k problems, 8 solutions each, 10 steps and 8 rollouts per step is 6.4 million completions, which is why OmegaPRM's binary search for the first error (roughly ${\log }_{2}K$ prefixes instead of $K$) matters.

Choices that matter#

Compute and memory#

Training a PRM costs about the same as training an ORM or SFT model of the same size on the same tokens: one forward and backward pass per solution, with the loss read at $K$ positions. Memory follows the usual rule of roughly 16 bytes per parameter for full fine-tuning with Adam, so a 7B PRM needs about 112 GB of model state before activations.

The expensive part is labels. Human step labels are slow; PRM800K's labellers had to check every step of competition math. MC labels replace that with generation: the 6.4 million completions in the example above, at a few hundred tokens each, are on the order of $10^9$ generated tokens. At inference, scoring is cheap (one forward pass per candidate), but best-of-N multiplies generation by $N$, and step-level beam search adds a PRM call per expansion. In RL, a PRM costs the same per rollout as an outcome reward model, since one pass scores all steps.

A toy experiment: label noise and aggregation#

How good are Monte Carlo labels?

We simulated 8-step solutions where each step is correct with probability $c$. From a clean prefix ending at step $k$, a completion succeeds with probability $c^{8-k}$. From a prefix containing an error, it still reaches the right answer with probability $\epsilon =0.1$ (a lucky guess or compensating error). We labelled every step with the hard MC rule (correct if any of $N$ rollouts succeeds) on 4,000 solutions and compared against the true labels (seed 0).

Hard MC labels trade one error for another as $N$ grows. More rollouts catch more good steps whose remaining path is hard (false "wrong" labels fall), but they also find the lucky completion after a real mistake (false "correct" labels follow $1-0.9^N$: 57% at $N=8$, 81% at $N=16$). On the easier problems ($c=0.9$) the best overall label accuracy was 0.86 at $N=4$. On the harder ones ($c=0.7$) no $N$ did well: accuracy peaked at 0.72 at $N=1$ and fell to 0.27 at $N=64$. Two lessons carry over to real PRMs. Hard labels from many rollouts systematically excuse errors on tasks where wrong reasoning can still hit the answer (multiple choice, small integer answers). And on hard problems, early steps get blamed for difficulty that lies later; that is the confusion between value and progress that Setlur et al. address.

Aggregation and outcome-only verifiers in best-of-16

Next we sampled solutions of 4 to 16 steps with a 10% per-step error rate; a flawed solution still had a 15% chance of a correct final answer. A simulated PRM produced noisy step probabilities (logit +1.5 for steps before the first error, -0.5 from the first error on, noise standard deviation 1.5). A simulated ORM saw only final-answer correctness through noise of the same size. We picked the best of 16 by each rule, over 3,000 problems.

On final-answer accuracy the ORM is competitive with the PRM (0.919 versus 0.935 for min). On process correctness it is far behind: 18% of its picks reached the right answer through a flawed derivation (0.919 minus 0.739), against 1.5% for the PRM with min aggregation. That is the Uesato et al. finding reproduced in miniature: final-answer metrics cannot tell the two verifiers apart, process metrics can. The product rule picked the shortest solutions (5.5 steps on average versus 10.0 sampled) and scored below min and mean. In this toy shorter solutions really are more often correct, so some of that preference is earned; on problems that need long derivations it becomes a bias against thorough answers. Mean slightly beat min here because our simulated PRM marks every step after an error as low, which a real PRM need not do, so we treat min as the safer default.

Failure modes and how to detect them#

Outcome drift. PRMs tuned for best-of-N accuracy learn to act like ORMs. Zhang et al. found that in many open PRMs the minimum step score concentrates on the final answer step. Check where the minimum score falls across a labelled set; if it is almost always the last step, you have an ORM with extra steps.

Inflated best-of-N numbers. Best-of-N final-answer accuracy rewards PRMs that accept correct answers with flawed processes. Evaluate step-level error localisation too: ProcessBench asks for the earliest erroneous step, which is the right test.

Biased MC labels. As the toy shows, hard labels excuse errors when wrong prefixes can still succeed. Use soft labels, progress labels or consensus with an LLM judge, and audit a few hundred labels by hand.

Reward hacking in RL. With raw step scores as dense rewards, policies learn to produce more, shorter, safer-looking steps, or to restate the problem in many trivially correct steps. Watch steps per solution and the ratio of PRM reward to final-answer accuracy; if PRM reward climbs while accuracy is flat, the policy is optimising the PRM, not the task. Progress-style rewards, or combining step rewards with a verifiable outcome reward, limit this.

Distribution shift. A PRM trained on GSM8K-level solutions fails to generalise to olympiad problems, as ProcessBench found for many open models. Validate on problems at the difficulty you will deploy on.

Segmentation mismatch. If the policy changes how it formats steps during RL (longer paragraphs, fewer newlines), the PRM sees step boundaries it was not trained on. Track steps per solution and enforce a step format.

When to use a PRM#

Minimal implementation sketch#

A PRM on top of a causal language model, with a step marker token and a scalar head read at each marker. The same model scores all steps in one forward pass.

import torch, torch.nn as nn, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

STEP = "<step>"
tok = AutoTokenizer.from_pretrained("path/to/base")
tok.add_special_tokens({"additional_special_tokens": [STEP]})
STEP_ID = tok.convert_tokens_to_ids(STEP)

class PRM(nn.Module):
    def __init__(self, name):
        super().__init__()
        self.body = AutoModel.from_pretrained(name, torch_dtype=torch.bfloat16)
        self.body.resize_token_embeddings(len(tok))
        self.head = nn.Linear(self.body.config.hidden_size, 1)
    def forward(self, input_ids, attention_mask):
        h = self.body(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        return self.head(h.float()).squeeze(-1)            # logits at every position [B, T]

def encode(problem, steps):
    text = problem + "\n" + "".join(s + STEP for s in steps)   # marker after each step
    return tok(text, return_tensors="pt", truncation=True, max_length=4096)

def prm_loss(model, batch_ids, batch_mask, step_labels):
    # step_labels: list of 1-D float tensors (1 = correct, 0 = error; soft values allowed),
    # one per example, truncated after the first error
    logits = model(batch_ids, batch_mask)
    losses = []
    for b, z in enumerate(step_labels):
        pos = (batch_ids[b] == STEP_ID).nonzero().squeeze(-1)[: len(z)]
        losses.append(F.binary_cross_entropy_with_logits(logits[b, pos], z.to(logits.device)))
    return torch.stack(losses).mean()

@torch.no_grad()
def score(model, problem, steps, agg="min"):
    enc = encode(problem, steps).to(next(model.parameters()).device)
    logits = model(enc["input_ids"], enc["attention_mask"])[0]
    p = torch.sigmoid(logits[enc["input_ids"][0] == STEP_ID])
    return {"min": p.min(), "prod": p.prod(), "mean": p.mean(), "last": p[-1]}[agg].item(), p.tolist()

For best-of-N, call score on each candidate and keep the highest; for weighted voting, sum scores per distinct final answer. In POFE a PRM is a reward-model recipe with a step-level evaluation report (error localisation on a held-out set, where the minimum score falls, and best-of-N curves) so outcome drift shows up before the PRM is used in RL.

Further reading#

Uesato et al. (2022) for the first process-versus-outcome comparison. Lightman et al. (2023) for PRM800K, active learning and the product aggregation. Wang et al. (2023) for Math-Shepherd's automatic labels and step-by-step PPO. Luo et al. (2024) for OmegaPRM's tree search labelling. Setlur et al. (2024) for progress-based process rewards. Zhang et al. (2025) and Zheng et al. (2024) for the limits of MC labels and step-level evaluation with ProcessBench. DeepSeek-AI (2025) for a frank account of why a large RL effort set PRMs aside.

Frequently asked questions#

References#

1. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins (2022). Solving math word problems with process- and outcome-based feedback. arxiv.org/abs/2211.14275
2. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe (2023). Let's Verify Step by Step. arxiv.org/abs/2305.20050
3. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, Zhifang Sui (2023). Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arxiv.org/abs/2312.08935
4. Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, et al. (2024). Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arxiv.org/abs/2406.06592
5. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, Aviral Kumar (2024). Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. arxiv.org/abs/2410.08146
6. Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, et al. (2025). The Lessons of Developing Process Reward Models in Mathematical Reasoning. arxiv.org/abs/2501.07301
7. Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, et al. (2024). ProcessBench: Identifying Process Errors in Mathematical Reasoning. arxiv.org/abs/2412.06559
8. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948

Related reading