GSPO (Group Sequence Policy Optimization) explained

Group Sequence Policy Optimization (GSPO) is a critic-free reinforcement-learning algorithm for language models that keeps GRPO's group-normalised advantage but moves the importance ratio, and therefore the clipping, from individual tokens to whole responses. Each sampled response gets one ratio, the length-normalised likelihood ratio between the current and the sampling policy, and the PPO-style clip either keeps or drops that response as a unit. It sits in the same slot as GRPO and DAPO: after SFT or a cold-start stage, training a reasoning model against verifiable rewards with several gradient steps per batch of rollouts. Its practical selling point is stability, especially for Mixture-of-Experts models, where token-level ratios become unreliable once expert routing shifts between the sampling and training passes.

Where GSPO came from#

GSPO was published in July 2025 by Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu and colleagues on the Qwen team at Alibaba (Zheng et al., 2025). The paper is short and opinionated. Its claim is that GRPO's instability when training very large models on long responses, often ending in collapse that the authors could not recover from even by reverting to an earlier checkpoint and retuning, is not bad luck or bad hyperparameters but a design error: GRPO "misapplies" importance sampling.

The argument runs as follows. Importance sampling corrects an expectation under a target distribution using samples from a behaviour distribution, and it works by averaging the reweighted samples over many draws. GRPO computes a ratio ${\pi }_{\theta }(y_t\mid x,y_{<t})/{\pi }_{{\theta }_{\text{old}}}(y_t\mid x,y_{<t})$ at every token position, but each position contributes exactly one sampled token from its next-token distribution. One draw cannot correct a distribution mismatch; it only multiplies that token's gradient by a random factor. Over thousands of tokens those factors add noise, and clipping, which zeroes some tokens and not others inside the same response, makes the noise worse. The authors' principle is that "the unit of optimization objective should match the unit of reward": the reward is given to the whole response, so the off-policy correction and the clipping should be applied to the whole response too.

The paper credits GSPO for the RL behind the later Qwen3 releases and reports two infrastructure wins. On Qwen3-30B-A3B, a 48-layer MoE model, about 10% of the experts activated for a given rollout change after a single gradient update, which made GRPO depend on a workaround called Routing Replay (caching the old routing and replaying it when computing the new probabilities). GSPO trains without it. And because a sequence-level likelihood is far more tolerant of small numerical differences than a per-token one, GSPO opens the door to using the inference engine's log-probabilities directly instead of recomputing them in the training engine.

The objective#

For a query $x$ from the dataset $\mathcal{D}$, sample a group of $G$ responses $y_1,\dots ,y_G$ from the old policy ${\pi }_{{\theta }_{\text{old}}}$. A verifier or reward model scores each one, $r(x,y_i)$. The advantage is GRPO's group-standardised reward, one scalar per response:

The new ingredient is the sequence-level importance ratio, defined as the geometric mean of the token ratios:

Here $|y_i|$ is the response length in tokens and ${\pi }_{\theta }(y_i\mid x)={\prod }_t{\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})$ is the full sequence likelihood. The GSPO objective, which is maximised, applies PPO's clipped surrogate once per response:

Compare GRPO, written in the same notation (KL term omitted, as the GSPO paper does):

The advantage is identical. What changes is where the ratio and the clip live: inside the token sum for GRPO, outside it for GSPO.

Why the ratio is length-normalised

The raw sequence ratio ${\pi }_{\theta }(y_i\mid x)/{\pi }_{{\theta }_{\text{old}}}(y_i\mid x)$ is a product of $|y_i|$ token ratios. For a 4,000-token response, a log-ratio of 0.001 per token compounds to $e^4\approx 55$; a few tokens moving a lot would swing it by orders of magnitude, and responses of different lengths would need different clip ranges. Taking the $|y_i|$-th root turns the product into the exponential of an average log-ratio, which keeps $s_i$ near 1 for every length and puts all responses on one scale. The cost is that $s_i$ is no longer an unbiased importance weight for the sequence; it is a trust-region signal, and GSPO uses it as one.

The gradient: equal weight for every token

Ignoring clipping, and using ${\nabla }_{\theta }{s}_i=s_i{\nabla }_{\theta }\log s_i$, the GSPO gradient is

while GRPO's is

This is the whole difference. In GRPO each token's score-function gradient is multiplied by its own ratio $w_{i,t}$, which can range over $(0,1+\epsilon ]$ for a positive advantage and $[1-\epsilon ,\infty )$ for a negative one. In GSPO every token in a response is multiplied by the same scalar $s_i$. A practical consequence the paper does not spell out: GSPO can rescale a response's gradient but never rotate it. Noise in the log-probabilities changes how much a response counts, not which of its tokens are pushed. With GRPO, the same noise reshuffles the relative weight of tokens inside the response.

GSPO-token, for per-token advantages

Multi-turn or tool-use training sometimes wants different advantages for different tokens. The paper defines a variant using a stop-gradient $\mathrm{sg}[\cdot ]$:

Numerically $s_{i,t}=s_i$, so the clip decision is still made per response, but the gradient of token $t$ is $s_i{\hat{A}}_{i,t}\nabla \log {\pi }_{\theta }(y_{i,t}\mid \cdot )$, which accepts a token-specific advantage. With ${\hat{A}}_{i,t}={\hat{A}}_i$ for all $t$ it is identical to GSPO in value, clipping and gradient. This is also the form most implementations use internally, because it lets a token-level loss tensor carry a sequence-level ratio.

Worked example: one response, both methods#

Take a four-token response. Under the sampling policy its tokens had probabilities $(0.50,0.40,0.90,0.04)$. After one minibatch update the current policy gives them $(0.58,0.30,0.90,0.05)$. The token ratios are

GSPO averages the log-ratios: $(0.1484-0.2877+0+0.2231)/4=0.02097$, so $s=e^{0.02097}=1.0212$. The response as a whole has become about 2.1% more likely per token.

Positive advantage, $\hat{A}=+1$. GRPO with $\epsilon =0.2$ clips token 4, whose ratio 1.25 is above 1.2, so that token stops contributing; tokens 1, 2 and 3 keep pushing with weights 1.16, 0.75 and 1.00 (divided by 4). GSPO with the paper's range $[1-3\times 10^{-4},1+4\times 10^{-4}]$ sees $s=1.0212>1.0004$ and drops the entire response from the gradient. The response has already moved far enough, by GSPO's standard.

Negative advantage, $\hat{A}=-1$. GRPO now clips on the low side: token 2, at 0.75, is below 0.8, so it is excluded, while tokens 1, 3 and 4 are pushed down with weights $-1.16/4$, $-1.00/4$ and $-1.25/4$. Token 4, which the policy has already raised by 25%, gets the largest push. GSPO's lower bound is $1-3\times 10^{-4}$, and $s=1.0212$ is above it, so the response is active, and every token receives the same weight $-1.0212/4=-0.2553$.

Two things are worth noticing. First, the clip ranges differ by roughly three orders of magnitude, which is not a typo: the average of many small log-ratios has a far narrower distribution than any single one. Second, GSPO's decision is coarser, all or nothing per response, and that is the point. The paper reports that GSPO clips about two orders of magnitude more tokens than GRPO and still learns faster, which the authors read as evidence that GRPO's token-level gradient contributions are mostly noise.

Why the clip range is so small#

Suppose each token's log-ratio carries independent noise with standard deviation $\sigma$, from routing changes, numerical differences between engines, or simply the stochastic update. Then $\log w_{i,t}$ has spread $\sigma$ at any length, while $\log s_i$, an average of $|y_i|$ such terms, has spread $\sigma /\sqrt{|y_i|}$. We checked this with 20,000 simulated responses per length at $\sigma =0.1$: the token log-ratio spread stayed at 0.100, and the sequence log-ratio spread fell from 0.0250 at 16 tokens to 0.0031 at 1,024 and 0.0016 at 4,096, exactly the $1/\sqrt{L}$ law (Figure 2). A fixed token-level range clipped 2.1% of tokens at every length.

The same simulation is a warning about copying hyperparameters. With $\sigma =0.1$ per token, more than 80% of even 4,096-token responses fall outside $[1-3\times 10^{-4},1+4\times 10^{-4}]$ on one side or the other. The paper's range works because, in its setting, per-token log-ratio drift between minibatches is far smaller than that. In yours it may not be. Measure the distribution of $\log s_i$ on your first few batches before fixing $\epsilon$.

The algorithm step by step#

for step in range(num_steps):
    prompts = sample_batch(dataset, B)
    with no_grad():
        groups  = [policy_old.generate(x, n=G) for x in prompts]       # B*G responses
        rewards = [[verifier(x, y) for y in g] for x, g in zip(prompts, groups)]
        adv     = [(r - mean(r)) / (std(r) + 1e-6) for r in rewards]    # one scalar per response
        logp_old = token_logprobs(policy_old, prompts, groups)         # or the inference engine's
    for mb in split(prompts, groups, adv, logp_old, n_minibatches=4):  # off-policy after the first
        logp  = token_logprobs(policy, mb)                              # (N, T)
        log_s = masked_mean(logp - mb.logp_old, mb.mask)                # (N,)  mean over tokens
        s     = exp(log_s)
        surr  = minimum(s * mb.adv, clip(s, 1 - eps_low, 1 + eps_high) * mb.adv)
        loss  = -surr.mean()                                            # one term per response
        loss.backward(); optimizer.step(); optimizer.zero_grad()
    policy_old.load_state_dict(policy.state_dict())

The rollout phase is GRPO's: sample $G$ responses per prompt, score them, standardise within the group. The difference is three lines in the loss. The first minibatch of every batch is exactly on-policy, so $s_i=1$, nothing is clipped, and GSPO's update equals GRPO's up to the token weighting (on the first minibatch both weightings are 1). The behaviour only diverges on the second and later minibatches, when ${\pi }_{\theta }$ has moved away from ${\pi }_{{\theta }_{\text{old}}}$. The paper splits each rollout batch into four minibatches; if you run fully on-policy, one gradient step per batch, GSPO and GRPO are the same algorithm.

Data it needs

The same as GRPO: prompts with whatever the reward function needs, such as reference answers for maths, test cases for code, or a reward model for open-ended tasks. GSPO adds no data requirements. The paper trained a cold-start model fine-tuned from Qwen3-30B-A3B-Base and evaluated on AIME'24, LiveCodeBench and CodeForces, regularly refreshing the query set and extending generation length during training. The usual advice carries over: filter prompts so that the starting policy's pass rate is spread across the middle of the range, because groups where every response gets the same reward have zero advantage under GSPO exactly as under GRPO.

Hyperparameters that matter#

The clip range is the only genuinely new knob, and it behaves differently from GRPO's. Because the sequence log-ratio concentrates as responses get longer, a fixed $\epsilon$ is effectively looser for short responses and tighter for long ones. Log three quantities per step: the fraction of responses clipped on each side, the mean and spread of $\log s_i$, and the same split by response length. If more than roughly half the responses with positive advantage are clipped from the second minibatch onward, you are wasting rollouts; widen ${\epsilon }_{\text{high}}$ or cut the learning rate. If almost nothing is ever clipped, the trust region is not doing anything and you might as well run fewer, larger steps.

Compute and memory#

GSPO costs the same as GRPO. The policy, an optional frozen reference (unused if $\beta =0$), the generation engine and a verifier are the only components; there is no critic. For full fine-tuning with Adam in mixed precision, a rough budget for an $N$-parameter policy is

with $2N$ more if you keep a reference model for a KL term. The loss itself adds one reduction over the token dimension. The savings are elsewhere. For MoE models, dropping Routing Replay removes the memory and communication needed to cache and replay expert assignments, and, as the authors note, removes a constraint that kept the model from using its full routing capacity during training. For asynchronous or disaggregated setups, the paper argues that sequence-level ratios are tolerant enough of numerical differences to use the inference engine's log-probabilities for ${\pi }_{{\theta }_{\text{old}}}$, saving a full forward pass over every rollout in the training engine. That claim is stated as a possibility, not measured, so validate it on your stack by comparing $\log s_i$ computed both ways.

Relative to DPO or SFT, GSPO has the cost profile of any online RL method: generation dominates, typically several times the cost of an SFT step on the same number of tokens.

A toy run with noisy log-probabilities#

We wanted to see whether the sequence-level ratio buys anything in a setting small enough to run in seconds and honest enough not to rig the result. The toy model: 64 prompts, each with a target string of $L$ tokens over a 6-token vocabulary, and an independent softmax per position (no shared parameters). The reward is the fraction of positions that match the target, so groups almost always have reward variance. Each step samples 16 prompts with $G=8$ responses, standardises rewards within the group, and takes four minibatch steps, so three of the four are off-policy. To mimic routing changes or engine mismatch we add independent Gaussian noise with standard deviation $\sigma$ to every behaviour log-probability before computing ratios. GRPO uses clip (0.2, 0.27) as in the paper's baseline; GSPO uses (0.0225, 0.03), scaled to this toy because the paper's $10^{-4}$ range clipped about 39% of responses here even without noise (and reached 0.728 instead of 0.764 at $L=32$). Learning rate 1,000 for both, 100 steps, 5 seeds; numbers are mean per-token accuracy, starting at 0.167.

Without noise the methods tie, which is what you would expect: in this toy every token has its own parameters, so token-level credit is actually correct and GRPO is not handicapped. With noise, both degrade, but GRPO degrades more, and the gap widens with length: 0.010 at 32 tokens, 0.019 at 128. That matches the mechanism in the gradient section. Noise rescales a GSPO response's update but leaves its direction alone, while under GRPO it reweights tokens within the response at random. The toy cannot show the dramatic, irreversible collapse the paper describes on a 30B MoE model. It has no shared parameters, no routing and no long horizon, so take it as evidence about the mechanism, not about the size of the effect at scale.

Failure modes and what to watch#

Everything clipped. If the fraction of clipped responses climbs towards 100% on minibatches two to four, GSPO has silently become one-step-per-batch training with three wasted passes. The cause is almost always a clip range copied from the paper into a setting with more per-token drift (higher learning rate, LoRA with a large rate, shorter responses). Plot the histogram of $\log s_i$ and set the range from it.

Short responses dominate the clip. Since the spread of $\log s_i$ shrinks with length, short responses are clipped more often for the same per-token drift. If response length correlates with correctness on your task, this skews which responses can learn. Log clip rates by length bucket.

A few tokens drive the ratio. One token whose probability moves from 0.001 to 0.1 contributes $\log 100\approx 4.6$ to the sum of log-ratios. In a 100-token response that alone shifts $\log s_i$ by 0.046, enough to clip the whole response under the paper's range. This is GSPO's version of outliers; GMPO (Zhao et al., 2025), a geometric-mean variant that clips per token in log space before averaging, is one response to it.

Inherited GRPO problems. GSPO does not change the advantage, so zero-variance groups, the std-division difficulty weighting discussed in Dr. GRPO, entropy collapse and reward hacking all apply unchanged. Track the zero-variance group rate, token entropy and a held-out score with a stricter grader.

Training-inference mismatch you did not measure. If you adopt the paper's suggestion of using inference-engine log-probabilities, a systematic bias between engines (not noise) shifts every $\log s_i$ in one direction and, with a range of $10^{-4}$, can clip one side of the advantage distribution wholesale. Compare the two log-probability sources on a sample before switching.

When to use GSPO#

Minimal implementation#

A PyTorch loss in the GSPO-token form, so the tensor shapes match a standard token-level trainer. Inputs are per-token log-probabilities and a completion mask.

import torch

def gspo_loss(logp, logp_old, adv, mask, eps_low=3e-4, eps_high=4e-4):
    """logp, logp_old, mask: (N, T); adv: (N,) group-normalised, one per response."""
    lengths = mask.sum(1).clamp(min=1)
    log_s = ((logp - logp_old) * mask).sum(1) / lengths          # (N,) mean token log-ratio
    # GSPO-token: value equals s_i, gradient flows through each token's logp
    log_s_tok = log_s.detach().unsqueeze(1) + logp - logp.detach()
    s = torch.exp(log_s_tok.clamp(max=10.0))
    a = adv.unsqueeze(1)
    surr = torch.minimum(s * a, torch.clamp(s, 1 - eps_low, 1 + eps_high) * a)
    per_seq = (surr * mask).sum(1) / lengths                      # every token equal weight
    clipped = ((s.detach() > 1 + eps_high) & (a > 0)) | ((s.detach() < 1 - eps_low) & (a < 0))
    stats = {"clip_frac_seq": clipped[:, 0].float().mean().item(),
             "log_s_mean": log_s.mean().item(), "log_s_std": log_s.std().item()}
    return -per_seq.mean(), stats

In TRL, GSPO is a configuration of GRPOTrainer. The library's paper index gives this reproduction of the paper's setting:

from trl import GRPOConfig

args = GRPOConfig(
    importance_sampling_level="sequence",   # the GSPO ratio
    loss_type="grpo",
    beta=0.0,
    epsilon=3e-4, epsilon_high=4e-4,
    gradient_accumulation_steps=1,
    steps_per_generation=4,                 # 4 minibatch updates per rollout batch
)

verl exposes it as actor_rollout_ref.actor.policy_loss.loss_mode=gspo with clip_ratio_low, clip_ratio_high and loss_agg_mode=seq-mean-token-mean, and OpenRLHF as --actor.policy_loss_type gspo. In POFE, switching a GRPO recipe to GSPO is a single setting, and the run dashboard shows the clip fraction by length so the range can be checked rather than assumed.

Further reading#

Zheng et al. (2025) is the primary source and is short enough to read in full. Shao et al. (2024) defines the GRPO objective GSPO modifies, and Schulman et al. (2017) the clipped surrogate both inherit. Yu et al. (2025) attacks the same long-sequence instability from the token side with DAPO's token-level loss and asymmetric clipping. Zhao et al. (2025) proposes GMPO, the closest sibling, which takes a geometric mean of clipped token ratios. MiniMax (2025), cited by the GSPO paper for the same collapse phenomenon, proposes CISPO, which clips the importance weight rather than the update.

Frequently asked questions#

References#

1. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, Junyang Lin (2025). Group Sequence Policy Optimization. arxiv.org/abs/2507.18071
2. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arxiv.org/abs/2402.03300
3. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov (2017). Proximal Policy Optimization Algorithms. arxiv.org/abs/1707.06347
4. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arxiv.org/abs/2503.14476
5. Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, et al. (2025). Geometric-Mean Policy Optimization. arxiv.org/abs/2507.20673
6. MiniMax (2025). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arxiv.org/abs/2506.13585
7. An Yang et al. (Qwen Team) (2025). Qwen3 Technical Report. arxiv.org/abs/2505.09388
8. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (GRPOTrainer, importance_sampling_level). github.com/huggingface/trl
9. ByteDance Seed and verl contributors (2024). verl: Volcano Engine Reinforcement Learning for LLMs (GSPO policy loss). github.com/volcengine/verl

Related reading