---

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS, 2025.

https://dapo-sia.github.io/

https://github.com/BytedTsinghua-SIA/DAPO

LLMRL1.9k+NeurIPS 2025CCF-A字节跳动清华大学香港大学

---

DAPO：大规模开源 LLM 强化学习系统

Abstract

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results.

推理扩展赋予 LLM 前所未有的推理能力，而强化学习是激发复杂推理的核心技术。 然而，最先进推理 LLM 的关键技术细节仍未公开，例如 OpenAI o1 博客和 DeepSeek R1 技术报告中的相关细节，因此社区仍难以复现其强化学习训练结果。

We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model.

我们提出解耦裁剪与动态采样策略优化（Decoupled Clip and Dynamic sAmpling Policy Optimization，DAPO）算法，并完整开源一个最先进的大规模强化学习系统；该系统以 Qwen2.5-32B 基础模型在 AIME 2024 上取得 50 分。

Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

不同于隐去训练细节的既有工作，我们介绍了使大规模 LLM 强化学习取得成功的四项关键算法技术。 此外，我们还开源了基于 verl 框架构建的训练代码，以及经过精心筛选和处理的数据集。 这些开源系统组件增强了可复现性，并为未来的大规模 LLM 强化学习研究提供支持。

图1：DAPO 在 Qwen2.5-32B 基础模型上的 AIME 2024 分数。DAPO 仅使用 50% 的训练步数便超过此前最先进的 DeepSeek-R1-Zero-Qwen-32B；横轴表示梯度更新步数。

1. Introduction

Test-time scaling such as OpenAI's o1 and DeepSeek's R1 brings a profound paradigm shift to Large Language Models (LLMs). Test-time scaling enables longer Chain-of-Thought thinking and induces sophisticated reasoning behaviors, which makes the models superior in competitive math and coding tasks like AIME and Codeforces.

OpenAI o1 和 DeepSeek R1 等测试时扩展方法为大语言模型（LLM）带来了深刻的范式转变。 测试时扩展支持更长的思维链思考并诱导复杂的推理行为，使模型在 AIME 和 Codeforces 等竞赛数学与编程任务上表现更优。

The central technique driving the revolution is large-scale Reinforcement Learning (RL), which elicits complex reasoning behaviors such as self-verification and iterative refinement. However, the actual algorithm and key recipe for scalable RL training remains a myth, hidden from technical reports of existing reasoning models. In this paper, we reveal significant obstacles in large-scale RL training and open-source a scalable RL system with fully open-sourced algorithm, training code and dataset that provides democratized solutions with industry-level RL results.

推动这场变革的核心技术是大规模强化学习（RL），它能够激发自我验证和迭代改进等复杂推理行为。 然而，可扩展强化学习训练所采用的实际算法与关键方案仍然成谜，现有推理模型的技术报告并未披露这些内容。 在本文中，我们揭示大规模强化学习训练中的重大障碍，并开源一个可扩展强化学习系统；其算法、训练代码和数据集均完整开放，以工业级强化学习结果提供普惠化解决方案。

We experiment over Qwen2.5-32B as the pretrained model for RL. In our initial GRPO run, we achieved only 30 points on AIME — a performance significantly below DeepSeek’s RL (47 points). A thorough analysis reveals that the naive GRPO baseline suffers from several key issues such as entropy collapse, reward noise, and training instability. The broader community has encountered similar challenges in reproducing DeepSeek's results suggesting that critical training details may have been omitted in the R1 paper that are required to develop an industry-level, large-scale, and reproducible RL system.

我们以 Qwen2.5-32B 作为强化学习的预训练模型开展实验。 在最初的 GRPO 训练中，我们在 AIME 上仅取得 30 分，显著低于 DeepSeek 强化学习得到的 47 分。 深入分析表明，朴素 GRPO 基线存在熵坍缩、奖励噪声和训练不稳定等若干关键问题。 更广泛的社区在复现 DeepSeek 结果时也遇到了类似挑战，这表明 R1 论文可能省略了构建工业级、大规模且可复现强化学习系统所必需的关键训练细节。

To close this gap, we release an open-source state-of-the-art system for large-scale LLM RL, which achieves 50 points on AIME 2024 based on Qwen2.5-32B model, outperforming previous state-of-the-art results achieved by DeepSeek-R1-Zero-Qwen-32B (47 points) using 50% training steps (Figure 1). We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and introduce 4 key techniques to make RL shine in the long-CoT RL scenario. Details are presented in Section 3.

为弥合这一差距，我们发布了一个最先进的开源大规模 LLM 强化学习系统；该系统基于 Qwen2.5-32B 模型在 AIME 2024 上取得 50 分，仅使用 50% 的训练步数便超过 DeepSeek-R1-Zero-Qwen-32B 此前取得的 47 分（图1）。 我们提出解耦裁剪与动态采样策略优化（Decoupled Clip and Dynamic sAmpling Policy Optimization，DAPO）算法，并引入四项关键技术，使强化学习在长思维链强化学习场景中充分发挥作用。 详细内容见第 3 节。

1. Clip-Higher, which promotes the diversity of the system and avoids entropy collapse;
2. Dynamic Sampling, which improves training efficiency and stability;
3. Token-Level Policy Gradient Loss, which is critical in long-CoT RL scenarios;
4. Overlong Reward Shaping, which reduces reward noise and stabilizes training.

1. 提高裁剪上界（Clip-Higher），它提升系统多样性并避免熵坍缩；
2. 动态采样（Dynamic Sampling），它提高训练效率与稳定性；
3. token 级策略梯度损失（Token-Level Policy Gradient Loss），它在长思维链强化学习场景中至关重要；
4. 超长奖励塑形（Overlong Reward Shaping），它减少奖励噪声并稳定训练。

Our implementation is based on verl. By fully releasing our state-of-the-art RL system including training code and data, we aim to reveal valuable insights to large-scale LLM RL that benefit the larger community.

我们的实现基于 verl。 通过完整发布包括训练代码和数据在内的最先进强化学习系统，我们希望揭示有关大规模 LLM 强化学习的宝贵见解，使更广泛的社区从中受益。

2. Preliminary

2.1 Proximal Policy Optimization (PPO)

PPO introduces a clipped surrogate objective for policy optimization. By constraining the policy updates within a proximal region of the previous policy using clip, PPO stabilizes training and improves sample efficiency. Specifically, PPO updates the policy by maximizing the following objective:

PPO 为策略优化引入了裁剪代理目标。 PPO 通过裁剪将策略更新约束在旧策略的邻近区域内，从而稳定训练并提高样本效率。 具体而言，PPO 通过最大化以下目标来更新策略：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{PPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}(q,a)\sim \mathcal{D},\\ o_{\le t}\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)\end{array}}[min(\frac{{\pi }_{\theta }(o_t\mid q,o_{<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_t\mid q,o_{<t})}{\hat{A}}_t,\\ & \mathrm{clip}(\frac{{\pi }_{\theta }(o_t\mid q,o_{<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_t\mid q,o_{<t})},1-\epsilon ,1+\epsilon ){\hat{A}}_t)].\end{array}$$

Here $(q,a)$ is a question-answer pair from the data distribution $\mathcal{D}$, $\epsilon$ is the clipping range of importance sampling ratio, and ${\hat{A}}_t$ is an estimator of the advantage at time step $t$. Given the value function $V$ and the reward function $R$, ${\hat{A}}_t$ is computed using the Generalized Advantage Estimation (GAE):

其中，$(q,a)$ 是来自数据分布 $\mathcal{D}$ 的问答对，$\epsilon$ 是重要性采样比率的裁剪范围，${\hat{A}}_t$ 是时间步 $t$ 的优势估计量。 给定价值函数 $V$ 和奖励函数 $R$，使用广义优势估计（GAE）计算 ${\hat{A}}_t$：

$${\hat{A}}_t^{\mathrm{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\mathrm{\infty }}(\gamma \lambda {)}^l{\delta }_{t+l}.$$

Here:

其中：

$${\delta }_l=R_l+\gamma V(s_{l+1})-V(s_l),0\le \gamma ,\lambda \le 1.$$

2.2 Group Relative Policy Optimization (GRPO)

Compared to PPO, GRPO eliminates the value function and estimates the advantage in a group-relative manner. For a specific question-answer pair $(q,a)$, the behavior policy ${\pi }_{{\theta }_{\text{old}}}$ samples a group of $G$ individual responses $\{o_i{\}}_{i=1}^G$. Then, the advantage of the $i$-th response is calculated by normalizing the group-level rewards $\{R_i{\}}_{i=1}^G$:

与 PPO 相比，GRPO 去除了价值函数，并以组相对方式估计优势。 对于特定问答对 $(q,a)$，行为策略 ${\pi }_{{\theta }_{\text{old}}}$ 采样一组 $G$ 个独立响应 $\{o_i{\}}_{i=1}^G$。 随后，通过对组级奖励 $\{R_i{\}}_{i=1}^G$ 进行归一化，计算第 $i$ 个响应的优势：

$${\hat{A}}_{i,t}=\frac{r_i-\mathrm{mean}(\{R_i{\}}_{i=1}^G)}{\mathrm{std}(\{R_i{\}}_{i=1}^G)}.$$

Similar to PPO, GRPO adopts a clipped objective, together with a directly imposed KL penalty term:

与 PPO 类似，GRPO 采用裁剪目标，并直接施加 KL 惩罚项：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{GRPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}(q,a)\sim \mathcal{D},\\ \{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)\end{array}}[\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\sum _{t=1}^{|o_i|}(min(r_{i,t}(\theta ){\hat{A}}_{i,t},\\ & \mathrm{clip}(r_{i,t}(\theta ),1-\epsilon ,1+\epsilon ){\hat{A}}_{i,t})-\beta D_{\mathrm{KL}}({\pi }_{\theta }\Vert {\pi }_{\mathrm{ref}}))].\end{array}$$

Here:

其中：

$$r_{i,t}(\theta )=\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}.$$

It is also worth noting that GRPO computes the objective at the sample-level. To be exact, GRPO first calculates the mean loss within each generated sequence, before averaging the loss of different samples. As we will be discussing in Section 3.3, such difference may have an impact on the performance of the algorithm.

还值得注意的是，GRPO 在样本级计算目标。 更确切地说，GRPO 首先计算每个生成序列内的平均损失，然后再对不同样本的损失取平均。 正如我们将在第 3.3 节中讨论的那样，这种差异可能会影响算法性能。

(a) AIME 准确率

(b) 参与者模型熵

图2：在强化学习训练过程中，应用 Clip-Higher 策略前后 AIME 测试集准确率与参与者模型生成概率熵的变化。

2.3 Removing KL Divergence

The KL penalty term is used to regulate the divergence between the online policy and the frozen reference policy. In the RLHF scenario, the goal of RL is to align the model behavior without diverging too far from the initial model. However, during training the long-CoT reasoning model, the model distribution can diverge significantly from the initial model, thus this restriction is not necessary. Therefore, we will exclude the KL term from our proposed algorithm.

KL 惩罚项用于调节在线策略与冻结参考策略之间的偏离。 在 RLHF 场景中，强化学习的目标是在不使模型与初始模型偏离过远的情况下对齐模型行为。 然而，在训练长思维链推理模型时，模型分布可能与初始模型显著不同，因此这一约束并无必要。 因此，我们将在所提出的算法中去除 KL 项。

2.4 Rule-based Reward Modeling

The use of reward model usually suffers from the reward hacking problem. Instead, we directly use the final accuracy of a verifiable task as the outcome reward, computed using the following rule:

使用奖励模型通常会遭遇奖励黑客问题。 因此，我们直接将可验证任务的最终准确性用作结果奖励，并按照以下规则计算：

$$R(\hat{y},y)=\{\begin{array}{ll}1,& \mathtt{\text{is\_equivalent}}(\hat{y},y),\\ -1,& \text{otherwise}.\end{array}$$

Here $y$ is the ground-truth answer and $\hat{y}$ is the predicted answer. This is proved to be an effective approach to activating the base model's reasoning capability, as shown in multiple domains such as automated theorem proving, computer programming, and mathematics competition.

其中，$y$ 是真实答案，$\hat{y}$ 是预测答案。 自动定理证明、计算机编程和数学竞赛等多个领域的结果表明，这是激活基础模型推理能力的一种有效方法。

3. DAPO

We propose the Decouple Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm. DAPO samples a group of outputs $\{o_i{\}}_{i=1}^G$ for each question $q$ paired with the answer $a$, and optimizes the policy via the following objective:

我们提出解耦裁剪与动态采样策略优化（Decouple Clip and Dynamic sAmpling Policy Optimization，DAPO）算法。 对于每个与答案 $a$ 配对的问题 $q$，DAPO 采样一组输出 $\{o_i{\}}_{i=1}^G$，并通过以下目标优化策略：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{DAPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}(q,a)\sim \mathcal{D},\\ \{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)\end{array}}[\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_i|}min(r_{i,t}(\theta ){\hat{A}}_{i,t},\\ & \mathrm{clip}(r_{i,t}(\theta ),1-{\epsilon }_{\mathrm{low}},1+{\epsilon }_{\mathrm{high}}){\hat{A}}_{i,t})],\\ \text{s.t.}& 0<|\{o_i\mid \mathtt{\text{is\_equivalent}}(a,o_i)\}|<G.\end{array}$$

Here:

其中：

$$\begin{array}{rl}{r}_{i,t}(\theta )& =\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})},\\ {\hat{A}}_{i,t}& =\frac{R_i-\mathrm{mean}(\{R_i{\}}_{i=1}^G)}{\mathrm{std}(\{R_i{\}}_{i=1}^G)}.\end{array}$$

The full algorithm can be found in Algorithm 1. In this section, we will introduce the key techniques associated with DAPO.

完整算法见算法1。 本节将介绍 DAPO 所采用的关键技术。

3.1 Raise the Ceiling: Clip-Higher

In our initial experiments using naive PPO or GRPO, we observed the entropy collapse phenomenon: the entropy of the policy decreases quickly as training progresses (Figure 2). The sampled responses of certain groups tend to be nearly identical. This indicates limited exploration and early deterministic policy, which can hinder the scaling process.

在使用朴素 PPO 或 GRPO 的初始实验中，我们观察到熵坍缩现象：随着训练推进，策略熵迅速下降（图2）。 某些组的采样响应趋于几乎完全相同。 这表明探索受到限制且策略过早变得确定，从而可能阻碍扩展过程。

We propose the Clip-Higher strategy to address this issue. Clipping over the importance sampling ratio is introduced in Clipped Proximal Policy Optimization (PPO-Clip) to restrict the trust region and enhance the stability of RL. We identify that the upper clip can restrict the exploration of the policy, where making an exploitation' token more probable is much easier yet the probability of an unlikely exploration' token is too tightly bounded to be uplifted.

我们提出 Clip-Higher 策略来解决这一问题。 裁剪近端策略优化（PPO-Clip）对重要性采样比率进行裁剪，以限制信赖域并增强强化学习稳定性。 我们发现，裁剪上界可能限制策略探索：提高“利用”token 的概率要容易得多，而低概率“探索”token 的概率却受到过紧约束，难以提升。

Concretely, when $\epsilon =0.2$ (the default value of most algorithms) and ${\hat{A}}_{i,t}>0$ (the system tries to increase the probability), consider two actions with probabilities ${\pi }_{{\theta }_{\mathrm{old}}}(o_i\mid q)=0.01$ and $0.9$. The upper bounds of the increased probabilities ${\pi }_{\theta }(o_i\mid q)$ are $0.012$ and $1.08$, respectively $({\pi }_{{\theta }_{\mathrm{old}}}\cdot (1+ϵ))$. This implies that `exploitation' tokens with a higher probability (e.g., 0.9) are not constrained to get even extremely larger probabilities like 0.999. Conversely, for low-probability `exploration' tokens, achieving a non-trivial increase in probability is considerably more challenging. Empirically, we also observe that the mean probability of up-clipped tokens is low: ${\pi }_{\theta }(o_i\mid q)<0.2$ (Figure 3). This finding supports our intuition that the upper clipping threshold indeed restricts the probability increase of low-probability `exploration' tokens, thereby potentially constraining the exploration of the system.

具体而言，当 $\epsilon =0.2$（大多数算法的默认值）且 ${\hat{A}}_{i,t}>0$（系统试图提高概率）时，考虑概率分别为 ${\pi }_{{\theta }_{\mathrm{old}}}(o_i\mid q)=0.01$ 和 $0.9$ 的两个动作。 提高后的概率 ${\pi }_{\theta }(o_i\mid q)$ 上界分别为 $0.012$ 和 $1.08$，即 $({\pi }_{{\theta }_{\mathrm{old}}}\cdot (1+ϵ))$。 这意味着，概率较高的“利用”token（例如 0.9）并未受到约束，其概率甚至可以进一步增大至 0.999。 相反，对于低概率“探索”token，要获得不可忽略的概率提升则困难得多。 在经验上，我们还观察到被上裁剪 token 的平均概率很低：${\pi }_{\theta }(o_i\mid q)<0.2$（图3）。 这一发现支持了我们的直觉：裁剪上界确实限制了低概率“探索”token 的概率提升，从而可能约束系统探索。

Adhering to the Clip-Higher strategy, we decouple the lower and higher clipping range as ${\epsilon }_{\text{low}}$ and ${\epsilon }_{\text{high}}$, as highlighted in the following equation:

按照 Clip-Higher 策略，我们将裁剪下界与上界解耦为 ${\epsilon }_{\text{low}}$ 和 ${\epsilon }_{\text{high}}$，如下式所示：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{DAPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}(q,a)\sim \mathcal{D},\\ \{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)\end{array}}[\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_i|}min(r_{i,t}(\theta ){\hat{A}}_{i,t},\\ & \mathrm{clip}(r_{i,t}(\theta ),1-{\epsilon }_{\mathrm{low}},1+{\epsilon }_{\mathrm{high}}){\hat{A}}_{i,t})],\\ \text{s.t.}& 0<|\{o_i\mid \mathtt{\text{is\_equivalent}}(a,o_i)\}|<G.\end{array}$$

We increase the value of ${\epsilon }_{\text{high}}$ to leave more room for the increase of low-probability tokens. As shown in Figure 2, this adjustment effectively enhances the policy's entropy and facilitates the generation of more diverse samples. We keep ${\epsilon }_{\text{low}}$ as it is, because increasing it will suppress the probability of these tokens to $0$, resulting in the collapse of the sampling space.

我们增大 ${\epsilon }_{\text{high}}$，为低概率 token 的概率提升留出更多空间。 如图2所示，这一调整有效提高了策略熵，并促进生成更多样的样本。 我们保持 ${\epsilon }_{\text{low}}$ 不变，因为增大它会将这些 token 的概率压低至 $0$，导致采样空间坍缩。

(a) 被上裁剪 token 的平均概率

(b) 准确率为 1 的样本比例

图3：被上裁剪 token 的平均概率，以及准确率为 1 的提示所占比例。

3.2 The More the Merrier: Dynamic Sampling

Existing RL algorithm suffers from the gradient-decreasing problem when some prompts have accuracy equal to 1. For example for GRPO, if all outputs $\{o_i{\}}_{i=1}^G$ of a particular prompt are correct and receive the same reward, the resulting advantage for this group is zero. A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient, thereby degrading sample efficiency. Empirically, the number of samples with accuracy equal to 1 continues to increase, as shown in Figure 3. This means that the effective number of prompts in each batch keeps decreasing, which can lead to larger variance in gradient and dampens the gradient signals for model training.

当某些提示的准确率等于 1 时，现有强化学习算法会遇到梯度减小问题。 以 GRPO 为例，如果某个提示的全部输出 $\{o_i{\}}_{i=1}^G$ 都正确并获得相同奖励，则该组得到的优势为 零。 零优势会产生零策略梯度，使批次梯度的幅度减小、对噪声更敏感，进而降低样本效率。 在经验上，准确率等于 1 的样本数量会持续增加，如图3所示。 这意味着每个批次中的有效提示数量不断减少，可能导致更大的梯度方差，并削弱模型训练的梯度信号。

To this end, we propose to over-sample and filter out prompts with the accuracy equal to 1 and 0 as illustrated in the following equation, leaving all prompts in the batch with effective gradients and keeping a consistent number of prompts. The sampling cost for each batch is dynamic. Before training, we keep sampling until the batch is fully filled with samples whose accuracy is neither 0 nor 1.

为此，我们提出过采样并过滤准确率等于 1 和 0 的提示，如下式所示，使批次中的所有提示都具有有效梯度，并保持提示数量一致。 每个批次的采样成本是动态的。 训练前，我们持续采样，直至批次被准确率既不为 0 也不为 1 的样本完全填满。

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{DAPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}(q,a)\sim \mathcal{D},\\ \{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)\end{array}}[\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_i|}min(r_{i,t}(\theta ){\hat{A}}_{i,t},\\ & \mathrm{clip}(r_{i,t}(\theta ),1-{\epsilon }_{\mathrm{low}},1+{\epsilon }_{\mathrm{high}}){\hat{A}}_{i,t})],\\ \text{s.t.}& 0<|\{o_i\mid \mathtt{\text{is\_equivalent}}(a,o_i)\}|<G.\end{array}$$

Note that this strategy does not necessarily impede training efficiency, because the generation time is typically dominated by the generation of long-tail samples if the RL system is synchronized and the generation stage is not pipelined. Besides, we find that with dynamic sampling the experiment achieves the same performance faster as shown in Figure 6.

需要注意的是，这一策略未必会降低训练效率，因为当强化学习系统采用同步方式且生成阶段未流水线化时，生成时间通常由长尾样本的生成主导。 此外，我们发现，采用动态采样后，实验能够更快达到相同性能，如图6所示。

3.3 Rebalancing Act: Token-Level Policy Gradient Loss

The original GRPO algorithm employs a sample-level loss calculation, which involves first averaging the losses by token within each sample and then aggregating the losses across samples. In this approach, each sample is assigned an equal weight in the final loss computation. However, we find that this method of loss reduction introduces several challenges in the context of long-CoT RL scenarios.

原始 GRPO 算法采用样本级损失计算：先在每个样本内部对逐 token 损失取平均，再跨样本聚合损失。 在这种方法中，每个样本在最终损失计算中被赋予相同权重。 然而，我们发现，这种损失归约方式在长思维链强化学习场景中带来了若干挑战。

Since all samples are assigned the same weight in the loss calculation, tokens within longer responses (which contain more tokens) may have a disproportionately lower contribution to the overall loss, which can lead to two adverse effects. First, for high-quality long samples, this effect can impede the model's ability to learn reasoning-relevant patterns within them. Second, we observe that excessively long samples often exhibit low-quality patterns such as gibberish and repetitive words. Thus, sample-level loss calculation, due to its inability to effectively penalize those undesirable patterns in long samples, leads to an unhealthy increase in entropy and response length, as shown in Figure 4.

由于损失计算为所有样本赋予相同权重，较长响应中包含的 token 更多，但其单个 token 对整体损失的贡献可能不成比例地更低，从而导致两种不良影响。 第一，对于高质量长样本，这种效应会妨碍模型学习其中与推理相关的模式。 第二，我们观察到，过长样本经常出现乱码和词语重复等低质量模式。 因此，样本级损失计算无法有效惩罚长样本中的这些不良模式，导致熵和响应长度以不健康的方式增长，如图4所示。

(a) 参与者模型生成概率的熵

(b) 参与者模型生成响应的平均长度

图4：参与者模型概率分布的熵以及响应长度的变化。

We introduce a Token-level Policy Gradient Loss in the long-CoT RL scenario to address the above limitations:

为解决上述局限，我们在长思维链强化学习场景中引入 token 级策略梯度损失：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{DAPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}(q,a)\sim \mathcal{D},\\ \{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)\end{array}}[\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_i|}min(r_{i,t}(\theta ){\hat{A}}_{i,t},\\ & \mathrm{clip}(r_{i,t}(\theta ),1-{\epsilon }_{\mathrm{low}},1+{\epsilon }_{\mathrm{high}}){\hat{A}}_{i,t})],\\ \text{s.t.}& 0<|\{o_i\mid \mathtt{\text{is\_equivalent}}(a,o_i)\}|<G.\end{array}$$

In this setting, longer sequences can have more influence on the overall gradient update compared to shorter sequences. Moreover, from the perspective of individual tokens, if a particular generation pattern can lead to an increase or decrease in reward, it will be equally prompted or suppressed, regardless of the length of the response in which it appears.

在这种设置下，相比短序列，长序列可以对整体梯度更新产生更大影响。 此外，从单个 token 的角度看，如果某种生成模式会导致奖励增加或减少，那么无论它出现在多长的响应中，都会受到同等程度的鼓励或抑制。

3.4 Hide and Seek: Overlong Reward Shaping

In RL training, we typically set a maximum length for generation, with overlong samples truncated accordingly. We find that improper reward shaping for truncated samples can introduce reward noise and significantly disrupt the training process.

在强化学习训练中，我们通常会设置生成长度上限，并相应截断过长样本。 我们发现，对截断样本进行不当的奖励塑形会引入奖励噪声，并严重干扰训练过程。

By default, we assign a punitive reward to truncated samples. This approach may introduce noise into the training process, as a sound reasoning process can be penalized solely due to its excessive length. Such penalties can potentially confuse the model regarding the validity of its reasoning process.

默认情况下，我们会为截断样本分配惩罚性奖励。 这种方法可能向训练过程引入噪声，因为一个合理的推理过程可能仅仅由于长度过长而受到惩罚。 这种惩罚可能使模型对自身推理过程是否有效产生困惑。

To investigate the impact of this reward noise, we first apply an Overlong Filtering strategy which masks the loss of truncated samples. We find that this approach significantly stabilizes training and enhances performance, as demonstrated in Figure 5.

为研究这种奖励噪声的影响，我们首先应用**超长过滤（Overlong Filtering）**策略，对截断样本的损失进行遮蔽。 我们发现，该方法显著稳定了训练并提升了性能，如图5所示。

(a) AIME 性能

(b) 参与者模型熵

图5：应用 Overlong Reward Shaping 策略前后，参与者模型在 AIME 上的准确率及其生成概率熵。

Algorithm 1: DAPO: Decoupled Clip and Dynamic sAmpling Policy Optimization

Input: initial policy model ${\pi }_{\theta }$; reward model $R$; task prompts $\mathcal{D}$; hyperparameters ${\epsilon }_{\mathtt{low}},{\epsilon }_{\mathtt{high}}$

1. for step $=1,\dots ,M$ do
2.   Sample a batch ${\mathcal{D}}_b$ from $\mathcal{D}$
3.   Update the old policy model ${\pi }_{{\theta }_{\mathrm{old}}}\leftarrow {\pi }_{\theta }$
4.   Sample $G$ outputs $\{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)$ for each question $q\in {\mathcal{D}}_b$
5.   Compute rewards $\{r_i{\}}_{i=1}^G$ for each sampled output $o_i$ by running $R$
6.   Filter out $o_i$ and add the remaining to the dynamic sampling buffer (Dynamic Sampling)
7.   if buffer size $n_b<N$:
8.     continue
9.   For each $o_i$ in the buffer, compute ${\hat{A}}_{i,t}$ for the $t$-th token of $o_i$
10.   for iteration $=1,\dots ,\mu$ do
11.     Update the policy model ${\pi }_{\theta }$ by maximizing the DAPO objective

Output: ${\pi }_{\theta }$

Furthermore, we propose Soft Overlong Punishment, a length-aware penalty mechanism designed to shape the reward for truncated samples. Specifically, when the response length exceeds the predefined maximum value, we define a punishment interval. Within this interval, the longer the response, the greater the punishment it receives. This penalty is added to the original rule-based correctness reward, thereby signaling to the model to avoid excessively long responses.

此外，我们提出软性超长惩罚（Soft Overlong Punishment），这是一种面向长度的惩罚机制，用于对截断样本的奖励进行塑形。 具体而言，当响应长度超过预定义最大值时，我们定义一个惩罚区间。 在该区间内，响应越长，受到的惩罚越大。 该惩罚会加到原始基于规则的正确性奖励上，从而向模型发出信号，使其避免生成过长响应。

$$R_{\mathrm{length}}(y)=\{\begin{array}{ll}0,& |y|\le L_{max}-L_{\mathrm{cache}},\\ {\displaystyle \frac{(L_{max}-L_{\mathrm{cache}})-|y|}{L_{\mathrm{cache}}}},& L_{max}-L_{\mathrm{cache}}<|y|\le L_{max},\\ -1,& L_{max}<|y|.\end{array}$$

3.5 Dataset Transformation

Our dataset is sourced from the web and official competition homepages through a combination of web scraping and manual annotation. The answers of math dataset typically come in a variety of formats, such as expression, formula and number, which makes it challenging to design comprehensive rules to parse them. To provide accurate reward signals using rules and minimize errors introduced by formula parsers, inspired by AIME, we select and transform the answers into integers, which are easy to parse. For example, if the original answer is expressed in the form of $\frac{a+\sqrt{b}}{c}$, we instruct the LLM to modify the question so that the expected answer becomes $a+b+c$. After selection and transformation, we obtained the DAPO-Math-17K dataset, which consists of 17K prompts, each paired with an integer as the answer.

我们的数据集通过网页抓取与人工标注相结合的方式，采集自互联网和官方竞赛主页。 数学数据集的答案通常具有表达式、公式和数值等多种格式，因此很难设计能够全面解析它们的规则。 受 AIME 启发，为了利用规则提供准确的奖励信号并尽量减少公式解析器引入的错误，我们筛选答案并将其转换为易于解析的整数。 例如，如果原始答案表示为 $\frac{a+\sqrt{b}}{c}$，我们会指示 LLM 修改问题，使预期答案变为 $a+b+c$。 经过筛选和转换后，我们得到 DAPO-Math-17K 数据集，它包含 17K 个提示，每个提示都与一个整数答案配对。

4. Experiments

4.1 Training Details

In this work, we focus specifically on mathematical tasks to evaluate our algorithm, which can be readily transferred to other tasks. We adopt the verl framework for training. We use naive GRPO as our baseline algorithm and estimate advantages using group reward normalization.

在本工作中，我们专门聚焦数学任务来评估算法，而该算法可以很容易地迁移到其他任务。 我们采用 verl 框架进行训练。 我们使用朴素 GRPO 作为基线算法，并通过组奖励归一化估计优势。

For hyper-parameters, we utilize the AdamW optimizer with a constant learning rate of $1\times {10}^{-6}$, incorporating a linear warm-up over 20 rollout steps. For rollout, the prompt batch size is 512 and we sample 16 responses for each prompt. For training, the mini-batch size is set to 512, i.e., 16 gradient updates for each rollout step. For Overlong Reward Shaping, we set the expected maximum length as 16,384 tokens and allocate additional 4,096 tokens as the soft punish cache. Therefore, the maximum number of tokens for generation is set to 20,480 tokens. As for the Clip-Higher mechanism, we set the clipping parameter ${\epsilon }_{\text{low}}$ to 0.2 and ${\epsilon }_{\text{high}}$ to 0.28, which effectively balance the trade-off between exploration and exploitation. For evaluation on AIME, we repeat the evaluation set for 32 times and report avg@32 for results stability. The inference hyperparameters of evaluation are set to temperature 1.0 and topp 0.7.

在超参数方面，我们使用 AdamW 优化器和恒定的 $1\times {10}^{-6}$ 学习率，并在 20 个 rollout 步内进行线性预热。 在 rollout 阶段，提示批大小为 512，并为每个提示采样 16 个响应。 在训练阶段，小批大小设为 512，即每个 rollout 步执行 16 次梯度更新。 对于超长奖励塑形，我们将预期最大长度设为 16,384 个 token，并额外分配 4,096 个 token 作为软惩罚缓存。 因此，生成 token 数量上限设为 20,480。 对于 Clip-Higher 机制，我们将裁剪参数 ${\epsilon }_{\text{low}}$ 设为 0.2，将 ${\epsilon }_{\text{high}}$ 设为 0.28，从而有效平衡探索与利用。 评估 AIME 时，我们将评估集重复 32 次，并报告 avg@32，以保证结果稳定性。 评估时的推理超参数设为温度 1.0、top-p 0.7。

图6：在基线设置中应用动态采样前后的训练进展。

4.2 Main Results

Experiments on AIME 2024 demonstrate that DAPO has successfully trained the Qwen-32B Base model into a powerful reasoning model, achieving performance superior to DeepSeek's experiments on Qwen2.5-32B using the R1 approach. In Figure 1, we observe a substantial improvement of performance on AIME 2024, with accuracy increasing from near $0$% to 50%. Notably, this improvement is achieved with only 50% of the training steps required by DeepSeek-R1-Zero-Qwen-32B.

AIME 2024 实验表明，DAPO 成功将 Qwen-32B Base 训练成一个强大的推理模型，其性能超过 DeepSeek 使用 R1 方法在 Qwen2.5-32B 上开展的实验。 在图1中，我们观察到 AIME 2024 性能大幅提升，准确率从接近 $0$% 提高至 50%。 值得注意的是，这一提升只使用了 DeepSeek-R1-Zero-Qwen-32B 所需训练步数的 50%。

We analyze the contributions of each training technique in our methodology, as detailed in Table 1. The observed improvements demonstrate the effectiveness of these techniques in RL training, each contributing several accuracy points in AIME 2024. Notably, given the vanilla GRPO setting, only 30% accuracy can be reached by training from a Qwen2.5-32B base model.

我们分析了方法中每项训练技术的贡献，详见表1。 观察到的提升证明这些技术在强化学习训练中有效，每项技术都为 AIME 2024 准确率贡献了数个百分点。 值得注意的是，在原始 GRPO 设置下，从 Qwen2.5-32B 基础模型开始训练只能达到 30% 的准确率。

For token-level loss, although it brings less performance improvement, we find it enhances training stability and makes the length increase more healthily.

对于 token 级损失，尽管它带来的性能提升较小，但我们发现它能增强训练稳定性，并使长度以更健康的方式增长。

When applying Dynamic Sampling, although more data needs to be sampled due to the filtering out of zero-gradient data, the overall training time is not significantly affected. As shown in Figure 6, although the number of sampling instances increases, the model's convergence time is even reduced, due to fewer training steps required.

应用动态采样时，由于过滤零梯度数据，需要采样更多数据，但整体训练时间并未受到显著影响。 如图6所示，尽管采样实例数量增加，但由于所需训练步数更少，模型的收敛时间反而缩短。

表1：逐步应用于 DAPO 的各项技术所取得的主要结果。

Model	AIME24avg@32
DeepSeek-R1-Zero-Qwen-32B	47
Naive GRPO	30
+ Overlong Filtering	36
+ Clip-Higher	38
+ Soft Overlong Punishment	41
+ Token-level Loss	42
+ Dynamic Sampling (DAPO)	50

4.3 Training Dynamics

Reinforcement learning on large language models is not only a cutting-edge research direction but also an intrinsically complex systems engineering challenge, characterized by the interdependence of its various subsystems. Modifications to any single subsystem can propagate through the system, leading to unforeseen consequences due to the intricate interplay among these components. Even seemingly minor changes in initial conditions, such as variations in data and hyperparameters, can amplify through iterative reinforcement learning processes, yielding substantial deviations in outcomes. This complexity often confronts researchers with a dilemma: even after meticulous analysis and well-founded expectations that a modification will enhance specific aspects of the training process, the actual results frequently diverge from the anticipated trajectory. Therefore, monitoring of key intermediate results during experimentation is essential for swiftly identifying the sources of discrepancies and, ultimately, for refining the system.

大语言模型强化学习不仅是前沿研究方向，也是一项本质上复杂的系统工程挑战，其特征是各子系统相互依赖。 对任一子系统的修改都可能在系统中传播，并由于这些组件之间复杂的相互作用而产生难以预见的后果。 即使是数据和超参数变化等看似微小的初始条件改变，也可能在迭代强化学习过程中被放大，导致结果出现显著偏差。 这种复杂性经常使研究人员陷入困境：即便经过细致分析，并有充分依据预期某项修改会改善训练过程的特定方面，实际结果仍常常偏离预期轨迹。 因此，在实验过程中监控关键中间结果，对于迅速确定偏差来源并最终改进系统至关重要。

(a) 平均响应长度

(b) 奖励分数

(c) 生成熵

(d) 平均概率

图7：DAPO 的响应长度、奖励分数、生成熵与平均概率曲线；这些曲线展示强化学习训练动态，并可作为识别潜在问题的重要监控指标。

• The Length of Generated Responses is a metric closely related to training stability and performance, as shown in Figure 7. The increase in length provides the model with a larger space for exploration, allowing more complex reasoning behaviors to be sampled and gradually reinforced through training. However, it is important to note that length does not always maintain a continuous upward trend during training. In some considerable periods, it can exhibit a trend of stagnation or even decline, which has also been demonstrated in DeepSeek-R1. We typically use length in conjunction with validation accuracy as indicators to assess whether an experiment is deteriorating.
• The Dynamics of Reward during training has always been one of the crucial monitoring indicators in reinforcement learning, as shown in Figure 7. In the majority of our experiments, the trend of reward increase is relatively stable and does not fluctuate or decline significantly due to adjustments in experimental settings. This indicates that, given a reliable reward signal, language models can robustly fit the distribution of training set. However, we find that the final reward on the training set often exhibits little correlation with the accuracy on the validation set, which indicates overfitting to the training set.
• The Entropy of the Actor Model and Generation Probability are related to the model's exploration capability and are key metrics that we closely monitor in our experiments. Intuitively, the model's entropy needs to be maintained within an appropriate range. An excessively low entropy indicates that the probability distribution is overly sharp, leading to a loss of exploration capability. Conversely, an excessively high entropy is often associated with issues of over-exploration such as gibberish and repetitive generation. For the generation probability, the situation is exactly the opposite. As demonstrated in Section 3.1, by applying the Clip-Higher strategy, we effectively addressed the issue of entropy collapse. In subsequent experiments, we find that maintaining a slow upward trend in entropy is conducive to the improvement of model performance, shown in Figure 7.

• 生成响应的长度是与训练稳定性和性能密切相关的指标，如图7所示。长度增加为模型提供了更大的探索空间，使其能够采样更复杂的推理行为，并通过训练逐步强化这些行为。然而，需要注意的是，训练期间长度并不总是持续上升。在一些相当长的时期内，它可能停滞甚至下降，DeepSeek-R1 也展示了这一现象。我们通常结合长度与验证准确率来判断实验是否正在恶化。
• 训练期间的奖励动态一直是强化学习中的关键监控指标之一，如图7所示。在我们的大多数实验中，奖励增长趋势相对稳定，不会因实验设置调整而显著波动或下降。这表明，只要奖励信号可靠，语言模型就能稳健拟合训练集分布。然而，我们发现，训练集上的最终奖励通常与验证集准确率相关性很低，这表明模型对训练集过拟合。
• 参与者模型熵与生成概率和模型的探索能力有关，是我们在实验中密切监控的关键指标。直观上，模型熵需要保持在适当范围内。熵过低表示概率分布过于尖锐，会导致探索能力丧失；相反，熵过高通常与乱码和重复生成等过度探索问题有关。生成概率的情况恰好相反。如第 3.1 节所示，应用 Clip-Higher 策略后，我们有效解决了熵坍缩问题。在后续实验中，我们发现，使熵保持缓慢上升趋势有利于提升模型性能，如图7所示。

4.4 Case Study

表2：强化学习中反思行为的涌现。

Question: Given that the base $ABC$ of the tetrahedron $S-ABC$ is an equilateral triangle, the projection $H$ of point $A$ on the face $SBC$ is the orthocenter of $\mathrm{△}SBC$, the dihedral angle $H-AB-C$ is ${30}^{\circ }$, and $SA=2$, find the volume of the tetrahedron. The answer is in the form $\frac{k}{m}$, so give the value of $k+m$.

Response: Let us consider the given problem: We need to determine ... ... Next, let $S$ have coordinates $(x_0,y_0,z_0)$. Since $SA=2$, the distance formula gives us $\sqrt{(x_0{)}^2+(y_0{)}^2+(z_0{)}^2}=2$, so we can write $x_0^2+y_0^2+z_0^2=4$. Now, remember that $H$ is the orthogonal projection of ... However, wait a moment, let's rethink about the dihedral angle involving planes in a more thoughtful geometric way. Consider the plane ${\alpha }_1=ABC$, which lies entirely on the xy coordinate plane (as $z=0$). Let the plane ${\alpha }_2=SBC$. The point $A$ projected perpendicularly to plane ${\alpha }_2$ lands on $H$. The line $l=AB$ ... ...

During the RL training process, we observe an interesting phenomenon: the reasoning patterns of the actor model evolve dynamically over time. Specifically, the algorithm not only reinforces existing reasoning patterns that facilitate correct problem-solving but also gradually gives rise to entirely new modes of reasoning that were initially absent. This finding reveals the adaptability and exploration capability of RL algorithms and offers new insights into the learning mechanisms of the model.

在强化学习训练过程中，我们观察到一个有趣现象：参与者模型的推理模式会随时间动态演化。 具体而言，该算法不仅强化有助于正确解决问题的既有推理模式，还会逐渐产生最初并不存在的全新推理方式。 这一发现揭示了强化学习算法的适应能力与探索能力，并为理解模型学习机制提供了新见解。

For example, in the early stages of model training, there was virtually no occurrence of checking and reflecting on previous reasoning steps. However, as training progresses, the model exhibits distinct behaviors of reflection and backtracking, as shown in Table 2. This observation sheds light on further exploration into interpreting the emergence of reasoning abilities during RL, which we leave for future research.

例如，在模型训练早期，几乎不会出现检查和反思先前推理步骤的行为。 然而，随着训练推进，模型会表现出明显的反思和回溯行为，如表2所示。 这一观察为进一步解释强化学习过程中推理能力的涌现提供了启示，我们将其留作未来研究。

5. Conclusion

In this paper, we release a fully open-sourced system for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The system achieves state-of-the-art large-scale LLM RL performance (AIME 50 using Qwen-32B pretrained model). We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and introduce 4 key techniques to make RL powerfully effective and efficient in the long-CoT RL scenario. Additionally, by open-sourcing the training code and dataset, we provide the broader research community and society with practical access to a scalable reinforcement learning solution, enabling all to benefit from these advancements.

在本文中，我们发布了一个完整开源的大规模 LLM 强化学习系统，包括算法、代码基础设施和数据集。 该系统实现了最先进的大规模 LLM 强化学习性能，即使用 Qwen-32B 预训练模型在 AIME 上取得 50 分。 我们提出解耦裁剪与动态采样策略优化（Decoupled Clip and Dynamic sAmpling Policy Optimization，DAPO）算法，并引入四项关键技术，使强化学习在长思维链强化学习场景中既强大有效又高效。 此外，通过开源训练代码和数据集，我们让更广泛的研究社区与社会能够实际使用可扩展强化学习解决方案，使所有人都能从这些进展中受益。