DAPO explained: decoupled clip and dynamic sampling policy optimisation

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) is a recipe for large-scale reinforcement learning on long chain-of-thought reasoning. It keeps GRPO's group-normalised advantage and drops the KL penalty, then adds four changes: an asymmetric clipping range that lets unlikely tokens grow faster ("clip-higher"), resampling until every group in the batch carries a learning signal ("dynamic sampling"), averaging the loss over all tokens in the batch rather than per response, and a graded penalty for responses that run past the length budget. It sits where GRPO sits, as the RL stage of reinforcement learning with verifiable rewards, usually applied directly to a strong base model. If GRPO is the algorithm, DAPO is the set of patches that made it hold up over thousands of steps with 20,000-token responses.

Where DAPO came from#

DAPO was published in March 2025 by Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo and 30 co-authors from ByteDance Seed, Tsinghua University and the University of Hong Kong, and accepted at NeurIPS 2025. The motivation was reproducibility. DeepSeek-R1 (DeepSeek-AI, 2025) had shown that GRPO on a base model with rule-based rewards produces strong reasoning, but key training details were missing, and the authors report that a naive GRPO run on Qwen2.5-32B reached only 30 points on AIME 2024 against the 47 reported for DeepSeek-R1-Zero-Qwen-32B. They diagnosed the gap as four separate failure modes: entropy collapse, wasted batches, mis-weighted long responses and noisy truncation penalties. Each technique fixes one.

The final system reached 50 points on AIME 2024 (avg@32) from the same base model, using half the training steps of the DeepSeek-R1-Zero-Qwen-32B result. The authors released the training code (built on the open-source verl framework; Sheng et al., 2024) and the DAPO-Math-17K dataset.

The objective#

For a question $q$ with reference answer $a$, sample $G$ responses $o_i$ from ${\pi }_{{\theta }_{\text{old}}}$. Each gets a rule-based reward, $R=1$ if the extracted answer is equivalent to $a$ and $R=-1$ otherwise. Define the per-token ratio $r_{i,t}(\theta )={\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})/{\pi }_{{\theta }_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})$ and the advantage, shared by every token of response $i$, as in GRPO:

DAPO maximises

Compare it term by term with GRPO's $\frac{1}{G}{\sum }_i\frac{1}{|o_i|}{\sum }_t\min (\cdot )-\beta {\mathbb{D}}_{\text{KL}}$. Four things changed: the normaliser is the total token count rather than a per-response average; the clip range has separate lower and upper bounds; there is no KL term; and the constraint says every group must contain at least one correct and at least one incorrect response. The fourth technique, overlong reward shaping, lives in the reward, not the objective.

The four techniques and why each works#

1. Clip-higher: decoupling the clip range

PPO's clip keeps the ratio $r_{i,t}$ inside $[1-\epsilon ,1+\epsilon ]$ for tokens whose advantage would push it further. The upper bound is a multiplicative cap on how much a token's probability can grow per batch. For a token at probability 0.9, a cap of $1.2$ means up to 1.08, which is no limit at all. For a token at 0.01, it means 0.012. The paper uses exactly this example. Clipping therefore protects exploitation of tokens the model already likes while throttling exploration of tokens it does not, and the authors observed the predictable result: policy entropy falling fast early in training, with groups converging to near-identical responses.

Clip-higher raises only the upper bound. With ${\epsilon }_{\text{low}}=0.2$ and ${\epsilon }_{\text{high}}=0.28$, a 0.01 token can reach 0.0128 per batch. Compounded, getting a 1% token to 50% under one maximal update per batch takes $\ln 50/\ln 1.2=21.5$ batches at $\epsilon =0.2$ and $\ln 50/\ln 1.28=15.8$ at 0.28, about a quarter fewer. The lower bound stays at 0.2 because lowering a token's probability toward zero is the move that actually collapses the distribution.

2. Dynamic sampling: no empty groups

With the group advantage, a group where all $G$ responses are correct, or all incorrect, has $\hat{A}=0$ for every token. As training progresses, more prompts become always-solved and the number of informative groups per batch falls, so the effective batch size shrinks and gradient noise grows. DAPO over-samples prompts and discards groups whose accuracy is exactly 0 or 1, generating more until the batch is full. That is the constraint in the objective. Because generation in long-CoT RL is dominated by the slowest (longest) responses, extra short groups often cost less wall-clock than the count suggests; the paper reports that the strategy does not necessarily slow training, and reached the same performance in fewer steps.

3. Token-level policy loss

GRPO first averages each response's per-token losses, then averages across responses. Every response therefore contributes equally regardless of length, so a token in a 2,000-token response weighs a tenth of a token in a 200-token one. DAPO normalises by the total number of tokens in the group (in practice, the batch). The paper names two consequences of the per-response average: long, high-quality reasoning gets too little credit per token, and long low-quality responses (repetition, gibberish) are not penalised enough per token, which shows up as unhealthy growth in length and entropy. It is the same diagnosis Dr. GRPO made a week later; the two fixes differ in the denominator.

4. Overlong filtering and soft overlong punishment

Responses that hit the generation cap are truncated, cannot reach a final answer, and get $R=-1$. That penalises a long but possibly correct line of reasoning as if it were wrong, which is reward noise. DAPO tried two remedies. Overlong filtering simply masks truncated samples out of the loss; it stabilised training. Soft overlong punishment adds a length-dependent term to the reward with an expected maximum $L_{\text{max}}-L_{\text{cache}}$ and a buffer $L_{\text{cache}}$:

DAPO set the expected maximum to 16,384 tokens and the buffer to 4,096, so generation is capped at 20,480. A correct 18,000-token answer receives $1+(16,384-18,000)/4,096=1-0.395=0.605$: still clearly better than a wrong answer, but with a gradient toward finishing sooner.

Why the KL term goes

The KL penalty in RLHF keeps the policy close to the reference so that it does not exploit a learned reward model. With a rule-based verifier there is no reward model to exploit, and the authors argue that a long-CoT reasoner is expected to move far from its initialisation, so the restriction is unnecessary. Dropping it also removes the reference model's forward pass and memory.

Worked example: two responses, two normalisations#

Take one group with two responses that both have advantage $\hat{A}$: one of 200 tokens and one of 2,000. Under GRPO's per-response average, each response gets weight $1/2$, so each token's weight is $1/(2\times 200)=0.0025$ in the short response and $1/(2\times 2000)=0.00025$ in the long one. The long response, with 91% of the tokens, carries 50% of the gradient. Under DAPO's token-level average, every token gets $1/2200=0.000455$, and the long response carries $2000/2200=91\%$ of the gradient. If the long response is a correct derivation, DAPO gives each of its tokens 1.8 times the weight GRPO did, and the same weight as a token in the short response instead of a tenth of it. If it is a repetitive failure, DAPO penalises each repeated token at the same rate as any other.

Now the clip. Suppose the long response is correct and contains a rare but pivotal token ("wait, let me re-check") at probability 0.02, with $\hat{A}=1.5$. With ${\epsilon }_{\text{high}}=0.2$, the surrogate $\min (r\hat{A},\mathrm{clip}(r)\hat{A})$ stops rewarding increases once $r>1.2$, so after this batch that token can be at most 0.024. With 0.28 it can reach 0.0256. If the same token appeared in a wrong response with $\hat{A}=-1.5$, both settings stop pushing it down below $0.8\times 0.02=0.016$.

Toy run: clip-higher and dynamic sampling#

To isolate the two sampling-side techniques we used a tabular policy: 512 prompts, each a softmax over 20 answers with one correct. A dominant wrong answer gets a +3 logit boost, and the correct answer's logit is shifted by a random amount in $[-1.5,2.5]$, so initial pass rates are low (mean 6.9%) and 231 prompts start below 3%. Each step takes 32 prompts with $G=16$ responses, rewards $\pm 1$, GRPO advantages, and four SGD passes over the batch with the clipped surrogate (so the clip actually binds). 150 steps, 5 seeds; all numbers are the script's output.

Clip-higher helped modestly and in the expected place: a 2.4-point gain overall and 2.2 points on the hardest prompts, with the share of positive-advantage samples hitting the clip falling from 32% to 25%. Dynamic sampling is more interesting. Per optimiser step it was a large win (70.3% versus 51.3%), because every group in the batch carried signal. Per rollout it was a wash: interpolated at the 76,800 rollouts the other runs used, it reached 50.8%, against 51.3% for clip-higher alone. In this toy, generating a rollout costs the same whatever its length, so dynamic sampling only buys you fewer, cleaner optimiser steps. In real long-CoT training, where step time is set by the longest response and optimiser steps carry fixed overheads, that trade is often favourable, which is the paper's argument. Measure it on your own wall-clock, not on step count.

The algorithm, step by step#

for step in range(num_steps):
    buffer = []
    while len(buffer) < B:                                   # dynamic sampling
        for q, a in sample_prompts(dataset, B_gen):
            outs = policy_old.generate(q, n=G, max_new_tokens=L_max)
            R = [ (1 if equivalent(extract(o), a) else -1) + overlong_penalty(len(o)) for o in outs ]
            acc = mean(equivalent(extract(o), a) for o in outs)
            if 0 < acc < 1:                                  # keep informative groups only
                buffer.append((q, outs, group_normalise(R)))
    batch = buffer[:B]
    for mb in minibatches(batch, size=mini_bs):              # several updates per batch
        ratio = exp(policy.logp(mb) - mb.logp_old)
        surr  = minimum(ratio * mb.adv, clip(ratio, 1 - eps_low, 1 + eps_high) * mb.adv)
        loss  = -(surr * mb.mask).sum() / mb.mask.sum()      # token-level: total tokens
        loss.backward(); opt.step(); opt.zero_grad()
    policy_old.load_state_dict(policy.state_dict())

Data. DAPO needs prompts with answers a program can check. The released DAPO-Math-17K set has 17K maths problems whose answers were transformed into integers so that equivalence checking is reliable; that preprocessing step is part of the method, because a verifier that rejects $\frac{1}{2}$ when the reference says 0.5 is a source of noisy $-1$ rewards. For other domains, the equivalent is unit tests for code or executable checks for SQL and structured extraction.

Hyperparameters#

The paper's ablation on Qwen2.5-32B shows how the pieces add up on AIME 2024: naive GRPO 30, plus overlong filtering 36, plus clip-higher 38, plus soft overlong punishment 41, plus token-level loss 42, plus dynamic sampling 50. The order is cumulative, so each number reflects one technique added on top of the earlier ones, not its effect in isolation.

Compute and memory#

Memory is GRPO's minus the reference model: for full fine-tuning with Adam, about 16 bytes per policy parameter for training state, plus a generation copy and its KV cache. For a 32B model that is roughly 512 GB of training state, which is why the original runs used multi-node setups with model sharding. The KV cache at 20,480 tokens dominates generation memory; at that length a single sequence of a 32B model with grouped-query attention can take several gigabytes of cache, and sixteen of them per prompt add up quickly.

Compute is dominated by generation, and dynamic sampling increases it by a factor of roughly $1/f$, where $f$ is the fraction of groups that are informative. In our toy that was about 1.8 times as many rollouts per step. Track $f$ directly: when it falls toward 0.2, five prompts are generated for every one used, and the dataset needs refreshing with harder problems. Compared with SFT on the same prompts, a DAPO step costs one to two orders of magnitude more, almost all of it in decoding long responses.

Failure modes and how to detect them#

Entropy collapse. Mean token entropy falling sharply in the first few hundred steps, followed by a rising share of zero-variance groups. That is what clip-higher is for; if it still happens, raise ${\epsilon }_{\text{high}}$ slightly or increase sampling temperature.

Entropy explosion. The opposite: entropy climbing and responses filling with gibberish or repetition. Usually ${\epsilon }_{\text{high}}$ set too high, a learning rate too large, or a missing token-level normaliser. Plot entropy and the fraction of responses with repeated n-grams together.

Resampling spiral. The informative-group rate $f$ falls steadily as the model masters the dataset, and generation cost per step rises with it. Refresh the prompt pool or move to harder problems.

Length runaway into the buffer. A growing share of responses landing between 16,384 and 20,480 tokens means the soft penalty is binding and the policy is trading accuracy for length. Check the mean penalty per batch; if it exceeds about 0.1, either the cap is too tight for the task or length is being rewarded somewhere else.

Verifier noise. Because there is no KL anchor, every verifier mistake is reinforced without restraint. Audit a sample of positive rewards by hand regularly, especially answers the parser accepted in an unusual format.

When to use DAPO#

Implementation#

The loss and the reward shaping, in PyTorch:

import torch

def overlong_penalty(length, L_expected=16384, L_cache=4096):
    L_max = L_expected + L_cache
    if length <= L_expected:
        return 0.0
    if length <= L_max:
        return (L_expected - length) / L_cache
    return -1.0

def dapo_loss(logp, logp_old, adv, mask, eps_low=0.2, eps_high=0.28):
    """logp, logp_old, mask: (N, T) per token; adv: (N,) group-normalised."""
    ratio = torch.exp(logp - logp_old)
    a = adv.unsqueeze(1)
    surr = torch.minimum(ratio * a, torch.clamp(ratio, 1 - eps_low, 1 + eps_high) * a)
    return -(surr * mask).sum() / mask.sum().clamp(min=1)   # token-level mean over the batch

def keep_group(correct_flags):
    k = sum(correct_flags)
    return 0 < k < len(correct_flags)                        # dynamic-sampling filter

Libraries now expose most of this as configuration. In TRL's GRPOConfig, loss_type="dapo" (the current default) gives token-level aggregation, epsilon=0.2 with epsilon_high=0.28 gives clip-higher, beta=0.0 removes the KL term, and mask_truncated_completions=True implements overlong filtering. Dynamic sampling and the soft length penalty usually need a custom reward function or sampler. The authors' own recipe is in the DAPO repository on top of verl. Whatever the implementation, log the informative-group rate and token entropy every step; they are the two signals DAPO's techniques are designed to move.

Further reading#

Yu et al. (2025) is the primary source and is unusually explicit about training dynamics; its entropy and length plots are worth studying before any long run. Read it with Shao et al. (2024) for the GRPO baseline it modifies and Liu et al. (2025) for the parallel critique of length normalisation. The same group later published VAPO (Yue et al., 2025), which brings a value model back for long-CoT tasks.

Frequently asked questions#

References#

1. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arxiv.org/abs/2503.14476
2. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arxiv.org/abs/2402.03300
3. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arxiv.org/abs/2501.12948
4. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arxiv.org/abs/2503.20783
5. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu (2024). HybridFlow: A Flexible and Efficient RLHF Framework. arxiv.org/abs/2409.19256
6. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, et al. (2025). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arxiv.org/abs/2504.05118
7. BytedTsinghua-SIA (2025). DAPO code and DAPO-Math-17K dataset. github.com/BytedTsinghua-SIA/DAPO
8. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arxiv.org/abs/1707.06347

Related reading