---

GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

Zhang J, Zuo C. GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models. arXiv preprint arXiv:2504.09696, 2025.

https://github.com/aeroplanepaper/GRPO-LEAD

LLMRLGRPO约翰斯·霍普金斯大学

---

GRPO-LEAD：面向语言模型简洁数学推理的难度感知强化学习方法

Abstract

Group Relative Policy Optimization (GRPO), which is widely adopted by R1-like reasoning models, has advanced mathematical reasoning. Nevertheless, GRPO faces challenges in reward sparsity, verbosity, and inadequate focus on problem difficulty. We propose GRPO-LEAD, enhancing GRPO with: (1) length-regularized rewards to encourage conciseness while maintaining accuracy; (2) explicit penalties for incorrect solutions to improve model precision; and (3) difficulty-aware advantage reweighting for robust generalization on challenging problems.

R1 类推理模型广泛采用的组相对策略优化（GRPO）推动了数学推理的发展。 然而，GRPO 仍面临奖励稀疏、输出冗长以及对问题难度关注不足等挑战。 我们提出 GRPO-LEAD，通过以下方式增强 GRPO：（1）使用长度正则化奖励，在保持准确率的同时鼓励简洁性；（2）对错误解答施加显式惩罚，以提高模型精确率；（3）采用难度感知的优势重加权，以增强模型在高难度问题上的稳健泛化能力。

Comprehensive evaluations demonstrate that GRPO-LEAD significantly improves reasoning accuracy, conciseness, and efficiency. Our approach achieves state-of-the-art performance for 14B-scale models, underscoring the synergy of our methods with appropriate model scale and high-quality data.

全面评估表明，GRPO-LEAD 显著提高了推理准确率、简洁性与效率。 我们的方法在 14B 规模模型上取得了当前最佳性能，凸显了这些方法与适当模型规模及高质量数据之间的协同作用。

1. Introduction

图1：GRPO-LEAD 框架为正确答案分配长度正则化的正奖励，并对错误答案施加显式惩罚。用于优势重加权的难度权重 w 由每个问题回答的经验正确率确定；该权重随后缩放各问题对应的优势，使策略更新更重视困难问题，从而促进稳健推理。

Recently, R1-like reasoning models have attracted significant attention due to their impressive performance in solving challenging mathematical reasoning tasks through extensive chains of thought. According to the technical report introducing R1, reinforcement learning (RL) fine-tuning plays a pivotal role in enabling this reasoning capability. In particular, Group Relative Policy Optimization (GRPO), a novel RL approach for language models, has emerged as a promising alternative to traditional methods such as PPO and DPO, primarily due to its efficiency and intrinsic compatibility with language model training. Researchers across various domains have successfully employed GRPO, achieving impressive outcomes.

近来，R1 类推理模型凭借通过长思维链解决高难度数学推理任务的出色表现，引起了广泛关注。 根据介绍 R1 的技术报告，强化学习（RL）微调在形成这种推理能力方面发挥了关键作用。 特别是，组相对策略优化（GRPO）作为一种面向语言模型的新型强化学习方法，主要凭借其高效性以及与语言模型训练的内在兼容性，成为 PPO 和 DPO 等传统方法的一种很有前景的替代方案。 不同领域的研究人员已成功采用 GRPO，并取得了令人瞩目的成果。

Despite its strengths, existing GRPO implementations encounter significant limitations. A primary issue is reward sparsity stemming from binary, rule-based accuracy metrics; when responses within problem groups exhibit uniform correctness or incorrectness, the resulting uniform reward signals offer minimal differentiation, weakening learning gradients and hampering convergence. Moreover, such uniform signals inadequately promote concise reasoning, leading to unnecessarily verbose outputs and inefficiencies during training and inference. Additionally, the current reward formulation lacks explicit penalties for incorrect answers, inadvertently encouraging models to guess rather than engage in rigorous reasoning, thereby compromising precision. Furthermore, rewards are applied uniformly across problems regardless of their intrinsic difficulty, causing models to excessively optimize simpler tasks while neglecting more challenging problems that require deeper reasoning.

尽管具有这些优势，现有 GRPO 实现仍存在显著局限。 一个主要问题是二元、基于规则的准确率指标造成的奖励稀疏：当同一问题组内的回答全部正确或全部错误时，统一的奖励信号几乎无法提供区分度，从而削弱学习梯度并阻碍收敛。 此外，这类统一信号无法充分鼓励简洁推理，导致输出不必要地冗长，并降低训练和推理效率。 同时，当前奖励形式没有对错误答案施加显式惩罚，无意中鼓励模型猜测而非进行严谨推理，因而损害精确率。 而且，无论问题本身难度如何，奖励都被统一应用于所有问题，导致模型过度优化简单任务，却忽视需要更深入推理的困难问题。

Furthermore, computational efficiency also emerges as a critical practical concern, as reinforcement learning fine-tuning typically demands substantial resources, limiting accessibility, experimentation speed, and scalability, especially in low-resource environments. The current GRPO formulation is insufficient for encouraging concise and precise reasoning. Consequently, reducing computational requirements during both training and inference is essential for enabling broader applicability and effective real-world deployment.

此外，计算效率也是一个关键的现实问题，因为强化学习微调通常需要大量资源，尤其在资源有限的环境中，这会限制其可及性、实验速度和可扩展性。 当前的 GRPO 形式不足以鼓励简洁而精确的推理。 因此，降低训练和推理阶段的计算需求，对于扩大适用范围和实现有效的现实部署至关重要。

Motivated by these limitations, this work introduces GRPO-LEAD, a suite of targeted modifications explicitly designed to enhance GRPO's effectiveness for mathematical reasoning tasks. The overall framework is illustrated in Figure 1. Our key contributions include:

受这些局限启发，本工作提出 GRPO-LEAD，这是一组专门设计的改进，旨在增强 GRPO 在数学推理任务上的有效性。 整体框架如 图1 所示。 我们的主要贡献包括：

• We introduce a length-regularized reward with an explicit penalty for incorrect solutions to encourage solution conciseness while maintaining accuracy.
• We apply difficulty-aware advantage reweighting to focus learning on more challenging problems, fostering robust generalization.
• Our comprehensive evaluations demonstrate GRPO-LEAD significantly improves reasoning accuracy and conciseness, achieving state-of-the-art performance in mathematical reasoning for 14B-scale models.

• 我们引入长度正则化奖励，并对错误解答施加显式惩罚，在保持准确率的同时鼓励解答简洁。
• 我们采用难度感知的优势重加权，使学习聚焦于更具挑战性的问题，从而促进稳健泛化。
• 我们的全面评估表明，GRPO-LEAD 显著提高了推理准确率与简洁性，并在 14B 规模模型的数学推理上取得当前最佳性能。

2. Related Work

2.1 Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) is a recently proposed algorithm designed specifically for fine-tuning language models with group-level normalization of rewards. GRPO modifies the standard policy gradient objective by introducing relative advantages within sets of responses corresponding to the same query, stabilizing updates and promoting consistent learning signals. Formally, GRPO defines the objective as:

组相对策略优化（GRPO）是近期提出的一种算法，专门用于通过组级奖励归一化对语言模型进行微调。 GRPO 在同一查询对应的回答集合内引入相对优势，从而修改标准策略梯度目标，稳定更新并促进一致的学习信号。 形式化地，GRPO 将目标定义为：

$$\begin{array}{rl}{\mathcal{L}}_{\mathrm{GRPO}}(\theta )& =\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\\ & \times \sum _{t=1}^{|o_i|}min(r_{i,t}(\theta ){\hat{A}}_{i,t},\\ & \mathrm{clip}\left(\begin{array}{c}{r}_{i,t}(\theta ),\\ 1-ϵ,1+ϵ\end{array}\right){\hat{A}}_{i,t}).\end{array}$$

The importance sampling ratio is given by:

重要性采样比率为：

$$r_{i,t}(\theta )=\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}.$$

Here, $G$ denotes the number of groups (e.g., different queries), ${\hat{A}}_{i,t}$ is the normalized advantage within group $i$, and $ϵ$ defines the clipping range for conservative updates.

其中，$G$ 表示组数（例如不同的查询），${\hat{A}}_{i,t}$ 是组 $i$ 内的归一化优势，$ϵ$ 定义用于保守更新的裁剪范围。

2.2 Length Reward

A prevalent issue in reinforcement learning-based fine-tuning of language models is reward hacking. In GRPO, when the model is trained with a large fixed budget, it can exploit this budget by producing an excessive number of extra reasoning and verification steps to ensure the correctness of the answer and therefore reach a higher reward. This phenomenon leads to unnecessarily verbose responses that lack conciseness and hinder interpretability, resulting in inefficiency in reasoning and reducing the model's practicality.

基于强化学习的语言模型微调普遍存在一个问题，即奖励破解。 在 GRPO 中，当模型使用较大的固定预算训练时，它可能通过生成过多额外的推理与验证步骤来利用这一预算，以确保答案正确并获得更高奖励。 这一现象会产生缺乏简洁性且不必要地冗长的回答，妨碍可解释性，造成推理低效并降低模型的实用性。

Existing efforts to mitigate this problem typically involve incentivizing shorter answers to encourage more succinct reasoning processes. For example, Kimi proposed an individual min-max normalized length reward based on the lengths of generated responses. Yeo et al. introduced a cosine length reward function with fixed maximum and minimum thresholds to manage response lengths. Aggarwal et al. utilized a target "golden length" to directly reward or penalize responses based on their deviation from an ideal length.

现有缓解方法通常通过激励更短的答案来鼓励更简洁的推理过程。 例如，Kimi 根据生成回答的长度提出了逐样本最小-最大归一化长度奖励。 Yeo 等人引入具有固定最大与最小阈值的余弦长度奖励函数，以管理回答长度。 Aggarwal 等人使用目标“黄金长度”，根据回答偏离理想长度的程度直接给予奖励或惩罚。

However, these existing methods depend heavily on static or predefined length heuristics, limiting their effectiveness across diverse questions of varying complexity. In contrast, our proposed length-dependent accuracy reward addresses these limitations by dynamically calibrating rewards according to each group’s relative response length and rollout accuracy, promoting concise yet difficulty-aware reasoning processes.

然而，这些现有方法高度依赖静态或预定义的长度启发式规则，限制了它们在复杂度各异的问题上的有效性。 相比之下，我们提出的长度依赖准确率奖励根据每个组的相对回答长度与 rollout 准确率动态校准奖励，从而促进既简洁又具难度感知能力的推理过程。

3. Method

To systematically address the limitations identified in existing implementations of Group Relative Policy Optimization (GRPO), we propose a suite of novel modifications collectively termed GRPO-LEAD (GRPO with Length-dependent rewards, Explicit penalties, and Advantage reweighting for Difficulty). Our proposed method enhances the original GRPO framework by introducing three core innovations: 1) a length-dependent accuracy reward to foster concise solutions, 2) an explicit penalty mechanism to mitigate low precision rate caused by length reward, and 3) a difficulty-aware advantage reweighting strategy that amplifies learning signals for challenging problems. Additionally, we examine how base model scale and supervised fine-tuning (SFT) impact the effectiveness of reinforcement learning (RL) fine-tuning.

为系统解决现有组相对策略优化（GRPO）实现中发现的局限，我们提出一组新的改进，统称为 GRPO-LEAD，即具有长度依赖奖励、显式惩罚以及面向难度的优势重加权的 GRPO。 我们的方法通过三项核心创新增强原始 GRPO 框架：1）使用长度依赖的准确率奖励促进简洁解答；2）通过显式惩罚机制缓解长度奖励造成的低精确率；3）使用难度感知的优势重加权策略，放大困难问题的学习信号。 此外，我们还研究基础模型规模与监督微调（SFT）如何影响强化学习（RL）微调的有效性。

3.1 Length-Dependent Accuracy Reward

The core idea is to reward correct completions not uniformly but in proportion to their relative conciseness. Given a question $q$ and a set of model-generated responses $\{o_i\}$, we first isolate the subset of correct responses and compute the mean $\mu$ and standard deviation $\sigma$ of their token lengths. For a correct response $o$ with length $|o|$, we define its standardized length deviation as:

核心思想不是统一奖励正确补全，而是按其相对简洁程度给予奖励。 给定问题 $q$ 和一组模型生成的回答 $\{o_i\}$，我们首先分离出正确回答子集，并计算其 token 长度的均值 $\mu$ 与标准差 $\sigma$。 对于长度为 $|o|$ 的正确回答 $o$，我们将其标准化长度偏差定义为：

$$z=\frac{|o|-\mu }{\sigma +ϵ},$$

Here, $ϵ>0$ is a small constant added for numerical stability. The final reward is modulated using an exponential decay function:

其中，$ϵ>0$ 是为保证数值稳定性而加入的小常数。 最终奖励使用指数衰减函数进行调制：

$$R_{\mathrm{accuracy}}(o\mid q)=\{\begin{array}{ll}\exp (-\alpha z),& \begin{array}{c}\text{if }o\text{ is}\\ \text{correct}\end{array},\\ 0,& \begin{array}{c}\text{if }o\text{ is}\\ \text{incorrect}\end{array}.\end{array}$$

Here, $\alpha >0$ is a tunable hyperparameter controlling the strength of length penalization. This formulation ensures that overly long correct responses are systematically penalized, while relatively concise ones are amplified. Unlike static or absolute length constraints, our approach leverages standardized deviation, allowing for dynamic adaptation to the distributional properties of each question.

其中，$\alpha >0$ 是控制长度惩罚强度的可调超参数。 这种形式确保过长的正确回答受到系统性惩罚，而相对简洁的回答则被放大。 与静态或绝对长度约束不同，我们的方法利用标准化偏差，可以动态适应每个问题的分布特性。

3.2 Explicit Penalty for Incorrect Answers to Enhance True Accuracy

Existing methods often prioritize maximizing pass@1—the success rate on the first attempt—typically within restricted response lengths. However, this focus can inadvertently degrade overall model accuracy. The fundamental issue appears to stem from the use of a binary accuracy reward, rather than length-based regularization: under pressure to generate responses within a limited length, a model is encouraged to provide an answer, even if it's a guess, rather than no answer at all. Such guesses can achieve a non-zero reward and inflate pass@1, but they do so at the cost of overall precision by rewarding less rigorous reasoning.

现有方法通常优先最大化 pass@1，即首次尝试的成功率，而且一般会限制回答长度。 然而，这种侧重可能无意中降低模型的整体准确率。 根本问题似乎来自二元准确率奖励，而非长度正则化：在有限长度内生成回答的压力下，模型会被鼓励给出一个答案，即使只是猜测，也好过完全不作答。 这类猜测可以获得非零奖励并抬高 pass@1，但它们通过奖励不够严谨的推理，以整体精确率为代价。

To counteract this tendency and foster a more robust distinction between correct and incorrect outputs, we introduce a revised reward structure that explicitly penalizes incorrect responses. This new reward function is defined as:

为抵消这一倾向，并在正确与错误输出之间形成更稳健的区分，我们引入一种修订后的奖励结构，对错误回答施加显式惩罚。 新的奖励函数定义为：

$$R_{\mathrm{accuracy}}(o\mid q)=\{\begin{array}{ll}\exp (-\alpha z),& \begin{array}{c}\text{if }o\text{ is}\\ \text{correct}\end{array},\\ -1,& \begin{array}{c}\text{if }o\text{ is}\\ \text{incorrect}\end{array}.\end{array}$$

Here, $o$ is the output, $q$ is the question, $z$ represents the standardized length deviation of a correct response, and $\alpha >0$ is a hyperparameter controlling the strength of the length penalization for correct answers, consistent with prior definitions. The expected reward for a response, given its probability of correctness $P(\mathrm{correct})$, under this formulation is:

其中，$o$ 是输出，$q$ 是问题，$z$ 表示正确回答的标准化长度偏差，$\alpha >0$ 是控制正确答案长度惩罚强度的超参数，与前述定义一致。 在这种形式下，给定回答的正确概率 $P(\mathrm{correct})$，其期望奖励为：

$$\begin{array}{rl}& \mathbb{E}[R_{\mathrm{accuracy}}(o\mid q)]\\ & =P(\mathrm{correct})\exp (-\alpha z)\\ & -(1-P(\mathrm{correct})).\end{array}$$

To intuitively grasp the impact of this reward function, let us consider a simplified scenario where the length penalty for correct answers is negligible (i.e., $\exp (-\alpha z)\approx 1$). In practice, the average reward for correct answers often normalizes close to this value. Under this assumption, the expected reward simplifies to:

为直观理解该奖励函数的影响，我们考虑一个简化情形：正确答案的长度惩罚可以忽略，即 $\exp (-\alpha z)\approx 1$。 在实践中，正确答案的平均奖励通常会归一化到接近这一数值。 在该假设下，期望奖励简化为：

$$\mathbb{E}[R]\approx 2P(\mathrm{correct})-1.$$

This approximation reveals a crucial characteristic: the expected reward becomes positive only when $P(\mathrm{correct})>0.5$. This threshold acts as a principled deterrent against speculative guessing, compelling the model to internalize a more stringent decision boundary for correctness. Our empirical results confirm that this approach significantly improves both pass@1 and overall precision, encouraging the model to favor accuracy over mere completion.

这一近似揭示了一个关键特性：只有当 $P(\mathrm{correct})>0.5$ 时，期望奖励才为正。 这一阈值从原理上抑制投机性猜测，迫使模型内化更严格的正确性决策边界。 我们的实证结果确认，该方法显著提高了 pass@1 与整体精确率，鼓励模型优先追求准确性，而非仅仅完成回答。

3.3 Advantage Reweighting for Difficulty-Aware Training

While length reward and advantage reweighting can enhance precision and mitigate verbosity, uniformly applying rewards across all questions, irrespective of their intrinsic difficulty, may implicitly bias the model. It might learn to excessively optimize performance on simpler tasks—where correct and concise responses are more readily achieved—while neglecting more complex questions that demand deeper reasoning. Consequently, the performance on challenging problems can degrade.

尽管长度奖励和优势重加权可以提高精确率并缓解冗长问题，但不考虑问题本身难度而对所有问题统一应用奖励，可能会给模型带来隐性偏差。 模型可能学会过度优化更简单的任务，因为这类任务更容易得到正确且简洁的回答，同时忽视需要更深入推理的复杂问题。 因此，模型在困难问题上的性能可能下降。

Therefore, we introduce a difficulty-aware advantage reweighting strategy, which dynamically adjusts the magnitude of policy updates based on an estimate of problem difficulty. The intuition is to amplify learning signals for harder tasks, re-anchoring the model towards harder tasks.

因此，我们引入难度感知的优势重加权策略，根据问题难度估计动态调整策略更新幅度。 其直觉是放大困难任务的学习信号，使模型重新聚焦于更困难的任务。

Formally, we first quantify problem difficulty. For a given question $q$ and its associated set of sampled responses $\{o_i\}$, we define the group's empirical correctness ratio as:

形式化地，我们首先量化问题难度。 对于给定问题 $q$ 及其对应的采样回答集合 $\{o_i\}$，我们将该组的经验正确率定义为：

$${\rho }_q=\frac{\text{number of correct responses for }q}{\text{total number of responses for }q}.$$

This ratio, ${\rho }_q$, serves as an inverse proxy for problem difficulty: a lower ${\rho }_q$ suggests a harder question. Next, we introduce a logistic reweighting factor dependent on this ratio to modulate the advantage estimates during the RL training step. The logistic function is defined as:

该比率 ${\rho }_q$ 可作为问题难度的反向代理：${\rho }_q$ 越低，问题越困难。 接下来，我们引入一个依赖该比率的逻辑重加权因子，在强化学习训练步骤中调制优势估计。 逻辑函数定义为：

$$w({\rho }_q)=A+\frac{B-A}{1+\exp [k({\rho }_q-{\rho }_0)]}.$$

Here, hyperparameters $A,B,{\rho }_0,k$ allow precise control over the sensitivity of weighting to problem difficulty. To apply this reweighting, we first consider the normalized advantage estimate for a response $o_i$ to question $q$:

其中，超参数 $A,B,{\rho }_0,k$ 可以精确控制权重对问题难度的敏感程度。 为应用这种重加权，我们首先考虑问题 $q$ 的回答 $o_i$ 所对应的归一化优势估计：

$${\tilde{A}}_i=\frac{R(o_i\mid q)-{\mu }_q}{{\sigma }_q+ϵ}.$$

Here, ${\mu }_q$ and ${\sigma }_q$ are the mean and standard deviation of rewards $R(o_i\mid q)$ for responses to question $q$, and $ϵ$ is a small constant for numerical stability. We then define the difficulty-aware advantage, $A_i^{\prime }$, as:

其中，${\mu }_q$ 和 ${\sigma }_q$ 分别是问题 $q$ 的回答奖励 $R(o_i\mid q)$ 的均值与标准差，$ϵ$ 是保证数值稳定性的小常数。 随后，我们将难度感知优势 $A_i^{\prime }$ 定义为：

$$A_i^{\prime }={\tilde{A}}_i\{\begin{array}{ll}w({\rho }_q),& \text{if }{\tilde{A}}_i>0,\\ w(1-{\rho }_q),& \text{if }{\tilde{A}}_i\le 0.\end{array}$$

This formulation ensures that for difficult problems (low ${\rho }_q$), correct responses (which are rare and thus highly valuable) receive substantially larger updates due to the increased weighting $w({\rho }_q)$. Conversely, incorrect responses on easier problems (high ${\rho }_q$) are penalized more strongly, sharpening the decision boundary for problems where high performance should be expected.

这种形式确保对于困难问题（低 ${\rho }_q$），正确回答由于稀少且价值很高，会在增大的权重 $w({\rho }_q)$ 下获得显著更大的更新。 相反，简单问题（高 ${\rho }_q$）上的错误回答会受到更强惩罚，从而为理应取得高性能的问题收紧决策边界。

3.4 Impact of Data Quality on Reinforcement Learning Effectiveness

To further enhance model capabilities, we first performed supervised fine-tuning (SFT) on a specialized dataset of 13k math reasoning problems sourced from DeepScaler (including historical AMC, AIME, and OmniMath problems) with solutions generated by QwQ32B. Although this SFT model initially showed signs of overfitting, subsequent application of our proposed RL strategies rapidly mitigated these issues. This SFT+RL approach yielded faster convergence and significantly improved pass@1 accuracy and overall precision compared to applying RL directly to the original base model.

为进一步增强模型能力，我们首先在一个包含 1.3 万道数学推理题的专门数据集上进行监督微调（SFT）；这些问题来自 DeepScaler，包括历史 AMC、AIME 和 OmniMath 题目，解答由 QwQ32B 生成。 尽管该 SFT 模型最初表现出过拟合迹象，随后应用我们提出的强化学习策略很快缓解了这些问题。 与直接在原始基础模型上应用强化学习相比，这种 SFT+RL 方法收敛更快，并显著提高了 pass@1 准确率和整体精确率。

Our findings also highlight the critical role of data quality and curriculum strategies in RL. We established a robust initial policy by applying RL to a subset of challenging problems from the DeepScaler dataset. This policy was then further refined using a curriculum composed of the most challenging problems identified from this first RL stage and supplemented by high-difficulty examples from the Light-R1 dataset. This two-stage curriculum markedly enhanced the model's ability to continuously improve on complex tasks.

我们的发现还凸显了数据质量和课程策略在强化学习中的关键作用。 我们在 DeepScaler 数据集的一部分困难问题上应用强化学习，以建立稳健的初始策略。 随后，我们使用一种课程进一步优化该策略：课程由第一阶段强化学习识别出的最困难问题组成，并补充 Light-R1 数据集中的高难度样本。 这种两阶段课程显著增强了模型在复杂任务上持续改进的能力。

Finally, we addressed a persistent formatting issue of repetitive n-gram patterns, likely stemming from an absence of clear end-of-sequence (EOS) signals during SFT. By temporarily removing length-dependent rewards and introducing an explicit negative reward ($-1.5$) for such repeated n-grams, we achieved further improvements in precision and pass@1 metrics. This intervention demonstrates the effectiveness of targeted reward modifications for mitigating specific output anomalies.

最后，我们处理了重复 n-gram 模式这一持续存在的格式问题，它很可能源于 SFT 期间缺少明确的序列结束（EOS）信号。 通过暂时移除长度依赖奖励，并对这类重复 n-gram 引入显式负奖励（$-1.5$），我们进一步提高了精确率和 pass@1 指标。 这一干预说明，有针对性地修改奖励可以有效缓解特定输出异常。

In summary, our experiments affirm that initial model capacity, curated data curricula for RL, and targeted reward engineering are pivotal for optimizing fine-tuning outcomes. These elements collectively inform a systematic approach for enhancing language models' ability to produce concise, accurate, and well-structured responses across tasks of varying complexity.

总之，我们的实验确认，初始模型能力、精心构建的强化学习数据课程和有针对性的奖励工程，是优化微调结果的关键。 这些要素共同构成一种系统方法，可以增强语言模型在不同复杂度任务上生成简洁、准确且结构良好回答的能力。

4. Experimental Setup

We evaluate GRPO-LEAD, integrating length-dependent accuracy rewards, explicit penalties for incorrect solutions, and difficulty-aware advantage reweighting, on DeepSeek-R1 Distilled variants. Our experiments cover two model scales, 7B and 14B parameters. All GRPO training is conducted using the VERL framework.

我们在 DeepSeek-R1 Distilled 系列模型上评估 GRPO-LEAD，该方法整合了长度依赖的准确率奖励、对错误解答的显式惩罚，以及难度感知的优势重加权。 实验覆盖 7B 和 14B 两种模型规模。 所有 GRPO 训练均使用 VERL 框架完成。

4.1 Datasets and Filtering

Our primary training data is sourced from the DeepScaler dataset. We filter out problems with difficulty ratings below 2.5, resulting in approximately 9,000 questions for fine-tuning.

我们的主要训练数据来自 DeepScaler 数据集。 我们过滤掉难度评分低于 2.5 的问题，最终得到约 9,000 道题用于微调。

For stage 2 of our 14B model experiments, we further refine the dataset by selecting problems where the model's stage-1 rollout accuracy is no greater than 75%, yielding around 2,283 questions. Additionally, we incorporate challenging problems with numeric answers from the stage-2 dataset of Light-R1, resulting in 3,524 questions in total.

在 14B 模型实验的第二阶段，我们进一步筛选第一阶段 rollout 准确率不高于 75% 的问题，得到约 2,283 道题。 此外，我们加入 Light-R1 第二阶段数据集中具有数值答案的困难问题，最终共得到 3,524 道题。

4.2 Hyperparameters

We train with a learning rate of $1\times {10}^{-6}$, batch size 32, and group size 8—generating 8 rollouts per question for GRPO reward computation. The KL penalty term is removed, as it was found to suppress exploration in our experiments, which is also suggested in similar works.

我们使用 $1\times {10}^{-6}$ 的学习率、32 的批大小和 8 的组大小进行训练，即每个问题生成 8 条 rollout 用于计算 GRPO 奖励。 我们移除了 KL 惩罚项，因为实验发现它会抑制探索，类似工作也提出了这一点。

For the length-dependent accuracy reward, we set $\alpha =0.05$, providing a moderate decay that encourages conciseness without penalizing slight verbosity. For difficulty-aware advantage reweighting, we use $A=0.4$, $B=1.5$, ${\rho }_0=0.75$, and $k=10$. This configuration ensures reweighting is minimal on easy problems but increases sharply near the 75% correctness threshold. The steep slope ($k=10$) enables strong emphasis on high-difficulty examples, guiding the model to allocate learning more effectively.

对于长度依赖的准确率奖励，我们设定 $\alpha =0.05$，以适度衰减鼓励简洁性，同时不惩罚轻微冗长。 对于难度感知的优势重加权，我们使用 $A=0.4$、$B=1.5$、${\rho }_0=0.75$ 和 $k=10$。 这一配置确保简单问题上的重加权很小，但在 75% 正确率阈值附近迅速增大。 较陡的斜率（$k=10$）能够强烈强调高难度样本，引导模型更有效地分配学习能力。

4.3 Model Variants and Fine-Tuning Stages

7B Model Experiments. Starting from the DeepSeek-R1 Distilled 7B Qwen-Math checkpoint, we first apply standard GRPO on the 9k questions, producing a baseline. Then, we train three more models from the DeepSeek-R1 Distilled 7B Qwen-Math checkpoint, adding one more of the following components subsequently: (i) Length Reward only, (ii) Length Reward + Advantage Reweighting, (iii) Length Reward + Advantage Reweighting + Explicit Penalty. We train for approximately 200 steps and select the top-performing checkpoints based on validation results. At test time, we limit the generation length to 8k for all 7B models, matching the training length limit.

7B 模型实验。 我们从 DeepSeek-R1 Distilled 7B Qwen-Math 检查点出发，首先在 9,000 道题上应用标准 GRPO，得到基线。 随后，我们仍从该检查点训练另外三个模型，并依次增加以下组件：（i）仅长度奖励；（ii）长度奖励 + 优势重加权；（iii）长度奖励 + 优势重加权 + 显式惩罚。 我们训练约 200 步，并根据验证结果选择表现最佳的检查点。 测试时，所有 7B 模型的生成长度均限制为 8k，与训练长度上限一致。

14B Model Experiments. We extend the above procedure to the DeepSeek-R1 Distilled 14B Qwen checkpoint across multiple stages. In Stage 1, we train for 100 steps using all GRPO-LEAD components on the filtered 9k-question dataset. To enhance the model's base capability, we first fine-tune the model on a curated set of 13k math problems with supervised fine-tuning (SFT), then conduct the RL phase. This SFT stage significantly improves the model’s reasoning quality, even though it tends to increase the output length and caused some format errors.

14B 模型实验。 我们将上述流程扩展到 DeepSeek-R1 Distilled 14B Qwen 检查点，并分多个阶段进行。 在 第一阶段，我们在筛选后的 9,000 道题上使用全部 GRPO-LEAD 组件训练 100 步。 为增强模型的基础能力，我们先在精心构建的 1.3 万道数学题上进行监督微调（SFT），然后进入强化学习阶段。 尽管该 SFT 阶段倾向于增加输出长度并造成一些格式错误，但它显著提高了模型的推理质量。

The SFT data consists of all problems in the DeepScaler dataset with difficulty greater than 1. To construct high-quality reasoning traces for SFT, we use the QWQ-32B model to generate step-by-step solutions.

SFT 数据由 DeepScaler 数据集中难度大于 1 的全部问题组成。 为了构建高质量的 SFT 推理轨迹，我们使用 QWQ-32B 模型生成逐步解答。

After observing that some questions remain low correctness, we further fine-tune for Stage 2 to focus on those underperformed problems. We also address the repetitive output patterns by removing the length penalty and introducing a negative reward ($-1.5$) for repeated $n$-grams. We continue training for 240 more steps (100 steps with initial settings and 140 more steps with repetition penalty), yielding the final model checkpoint. At test time, we limit the generation length to 14k for all 14B models, in accordance with our training settings and also to better compare the models' performance in a low-budget scenario.

在观察到一些问题仍然具有较低正确率后，我们进一步进行 第二阶段 微调，聚焦于这些表现不佳的问题。 我们还通过移除长度惩罚，并对重复的 $n$-gram 引入负奖励（$-1.5$），来处理重复输出模式。 我们继续训练 240 步，其中 100 步使用初始设置，另外 140 步使用重复惩罚，最终得到模型检查点。 测试时，所有 14B 模型的生成长度均限制为 14k，这既符合训练设置，也有助于在低预算场景中更好地比较模型性能。

4.4 Baselines and Evaluation Protocol

We compare our models with both DeepSeek-R1 Distilled-14B-Qwen (the distilled Qwen model without GRPO-LEAD) and Light-R1-14B-DS, which has the same base model as ours and was first finetuned with 3k hard math problems with SFT, and then fine-tuned with a cosine-based length reward on their selected math problems for three epochs using GRPO.

我们将模型与 DeepSeek-R1 Distilled-14B-Qwen（未使用 GRPO-LEAD 的蒸馏 Qwen 模型）和 Light-R1-14B-DS 进行比较；后者与我们采用相同的基础模型，先使用 3,000 道困难数学题进行 SFT，再在选定的数学题上结合基于余弦的长度奖励，使用 GRPO 微调三个 epoch。

We primarily report three metrics: (1) Cons@32, accuracy through majority voting for 32 samplings; (2) Pass@1, the probability that the top-1 sample is correct under a chosen decoding strategy; (3) Average Length (Len${}_{\mathrm{avg}}$), measuring verbosity. Unless otherwise specified, we decode with temperature 0.6 and sample 32 solutions per question, then compute Cons@32 and Pass@1 over these samples.

我们主要报告三个指标：（1）Cons@32，对 32 次采样进行多数投票得到的准确率；（2）Pass@1，在选定解码策略下排名第一的样本正确的概率；（3）平均长度（Len${}_{\mathrm{avg}}$），用于衡量冗长度。 除非另有说明，我们使用温度 0.6 进行解码，每个问题采样 32 个解答，再基于这些样本计算 Cons@32 和 Pass@1。

5. Results

表1：AIME24 和 AIME25 上的消融结果。我们报告 Cons@32（32 次采样多数投票的准确率）、Pass@1 和平均 token 长度（Len_avg）。每列最佳值以粗体表示，次佳值以下划线表示。

Ablation Setting	AIME24			AIME25
	Cons@32	Pass@1	Lenavg	Cons@32	Pass@1	Lenavg
Deepseek-7B	0.767	0.431	6,990	0.467	0.292	7,113
GRPO + len. reward	0.767	0.438	5,275	0.533	0.308	5,210
+ adv. reweighting	0.767	0.458	5,323	0.567	0.325	5,437
+ explicit penalty	0.800	0.470	6,104	0.567	0.345	6,308

In this section, we present a comprehensive evaluation of the proposed GRPO-LEAD framework on two mathematical benchmarks: AIME24 and AIME25. Our analysis is structured as follows: we first examine training dynamics to illustrate how GRPO-LEAD accelerates convergence; next, we perform an ablation study to assess the incremental benefits of each component; and finally, we compare against state-of-the-art baselines for 14B-scale language models.

本节在 AIME24 和 AIME25 两个数学基准上，对所提出的 GRPO-LEAD 框架进行全面评估。 我们的分析安排如下：首先考察训练动态，以说明 GRPO-LEAD 如何加速收敛；接着进行消融研究，以评估各组件带来的增量收益；最后与 14B 规模语言模型的当前最佳基线进行比较。

5.1 Training Dynamics

图2：三种配置在训练过程中的验证集 Pass@1：GRPO、GRPO+L 和 GRPO+LAD。更快的收敛表明，长度奖励与优势重加权提供了比原始设置更丰富的奖励信号。

^* 验证集由 AIMO2、CMU-MATH-AIMO 和 AIME24 中的 27 道困难问题组成。

Figure 2 plots the evolution of Pass@1 on a validation split over training steps for three configurations of the 7B model: (i) baseline GRPO, (ii) GRPO with length reward, and (iii) GRPO with both length reward and advantage reweighting. We observe two clear trends. First, adding a length-dependent reward not only yields higher Pass@1 but also accelerates early-stage convergence, suggesting that penalizing overly verbose correct solutions provides a more informative learning signal. Second, incorporating advantage reweighting (to amplify updates on harder questions) further steepens the trajectory, indicating that reweighting advantage estimates according to problem difficulty helps the model refine reasoning on challenging prompts more efficiently.

图2 绘制了 7B 模型三种配置在训练过程中验证集 Pass@1 的变化：（i）基线 GRPO；（ii）加入长度奖励的 GRPO；（iii）同时加入长度奖励与优势重加权的 GRPO。 我们观察到两个清晰趋势。 第一，加入长度依赖奖励不仅提高了 Pass@1，还加速了训练早期的收敛，这表明惩罚过度冗长的正确解答可以提供信息量更高的学习信号。 第二，加入优势重加权以放大困难问题上的更新后，曲线进一步变陡，说明根据问题难度重加权优势估计，有助于模型更高效地改进困难提示上的推理。

Overall, these dynamics confirm that GRPO-LEAD components---particularly the length reward---bolster training stability and speed. By comparison, the baseline GRPO model learns more slowly and lags behind in Pass@1 across the entire training horizon.

总体而言，这些动态确认，GRPO-LEAD 的组件，尤其是长度奖励，可以增强训练稳定性并提高训练速度。 相比之下，基线 GRPO 模型学习更慢，并且在整个训练过程中始终落后于其他配置的 Pass@1。

5.2 Ablation Analysis

(a) AIME24

(b) AIME25

图3：不同 LEAD 消融配置训练后的性能与推理预算关系。与训练前的 DeepseekR1-7B 相比，加入长度奖励的 GRPO（GRPO+L）在低预算设置下显著提高了性能。

We next quantify the contribution of each GRPO-LEAD component through a step-by-step ablation on the 7B model. Table 1 summarizes results on AIME24 and AIME25.

接下来，我们通过在 7B 模型上逐步进行消融，量化 GRPO-LEAD 各组件的贡献。 表1 汇总了 AIME24 和 AIME25 上的结果。

Length Reward Brings Conciseness to Reasoning. We first incorporate the length-dependent accuracy reward into GRPO. Compared to Deepseek-7B, length reward slightly improves Pass@1 on both AIME24 by 1.6% ($0.431\to 0.438$) and AIME25 by 5.4% ($0.292\to 0.308$), with an additional improvement of Cons@32 by 14.1% on AIME25. Notably, these improvements are accompanied by a substantial reduction of 1,715 tokens (24.5%) and 1,903 tokens (26.8%) in the average response length on the two datasets, respectively. Figure 3 further demonstrates that length reward largely enhances performance in low-budget settings over the base model, matching its peak performance with only 5/8 of the token budget on the more difficult AIME25. These results demonstrate that length reward, by penalizing correct but overly verbose solutions, can effectively reduce unnecessary text without compromising overall performance.

长度奖励使推理更加简洁。 我们首先将长度依赖的准确率奖励加入 GRPO。 与 Deepseek-7B 相比，长度奖励使 AIME24 上的 Pass@1 提高 1.6%（$0.431\to 0.438$），使 AIME25 上的 Pass@1 提高 5.4%（$0.292\to 0.308$），并使 AIME25 上的 Cons@32 进一步提高 14.1%。 值得注意的是，这些改进还分别伴随两个数据集平均回答长度减少 1,715 个 token（24.5%）和 1,903 个 token（26.8%）。 图3 进一步表明，相较基础模型，长度奖励显著提高了低预算设置下的性能；在更困难的 AIME25 上，它只使用 5/8 的 token 预算便达到基础模型的峰值性能。 这些结果表明，长度奖励通过惩罚正确但过度冗长的解答，可以在不损害整体性能的情况下有效减少不必要的文本。

Advantage Reweighting Encourages Model to Solve More Difficult Problems. Further incorporating difficulty-aware advantage reweighting (GRPO+LAD) refines performance. On AIME24, Pass@1 increases from the GRPO+L stage by 4.8% ($0.438\to 0.458$), while Cons@32 remains $0.767$. For AIME25, both Pass@1 and Cons@32 improve by 5.5% ($0.308\to 0.325$) and 6.4% ($0.533\to 0.567$), respectively. As Figure 3 shows, GRPO+LAD demonstrates gains over GRPO+L in almost all budget regimes on AIME25 and for budgets exceeding 5k tokens on AIME24. These results indicate that advantage reweighting, by prioritizing challenging problems, strengthens reasoning robustness and mitigates over-reliance on simpler examples, thus validating its role in driving more reliable generalization.

优势重加权鼓励模型解决更困难的问题。 进一步加入难度感知的优势重加权（GRPO+LAD）后，性能得到改善。 在 AIME24 上，Pass@1 相较 GRPO+L 阶段提高 4.8%（$0.438\to 0.458$），Cons@32 则保持在 $0.767$。 在 AIME25 上，Pass@1 和 Cons@32 分别提高 5.5%（$0.308\to 0.325$）和 6.4%（$0.533\to 0.567$）。 如 图3 所示，在 AIME25 的几乎所有预算范围内，以及 AIME24 中超过 5k token 的预算下，GRPO+LAD 均优于 GRPO+L。 这些结果表明，优势重加权通过优先处理困难问题，增强了推理稳健性并缓解对简单样本的过度依赖，从而验证了它在促进更可靠泛化方面的作用。

表2：AIME24 和 AIME25 上的模型性能比较，包括 Cons@32、Pass@1 和平均 token 长度（Len_avg）。每列最佳值以粗体表示，次佳值以下划线表示。

Model Name	AIME24			AIME25
	Cons@32	Pass@1	Lenavg	Cons@32	Pass@1	Lenavg
DeepSeek-14B	0.800	0.614	9,182	0.633	0.429	10,046
Light-R1-14B-DS	0.833	0.641	9,571	0.767	0.505	10,194
LEAD-stage1	0.833	0.629	8,790	0.767	0.523	9,371
LEAD-stage2	0.867	0.650	8,267	0.767	0.539	8,668

Explicit Penalty for Incorrect Answers Regularizes Thinking. Finally, introducing an explicit penalty for incorrect solutions (GRPO+LEAD) yields the highest Pass@1 scores. On AIME24, Pass@1 and Cons@32 improve from the GRPO+LAD stage by 2.6% ($0.458\to 0.470$) and 4.3% ($0.767\to 0.800$), respectively. On AIME25, Pass@1 also increases by 6.2% ($0.325\to 0.345$), as detailed in Table 1. Notably, these gains involve a modest increase in average solution length on AIME24 (from approximately 5,300 to 6,104 tokens). Figure 3 illustrates this trade-off, showing a performance sacrifice in low-budget regimes, though GRPO+LEAD still outperforms GRPO+LAD with budgets higher than 5k tokens on AIME25. These results suggest that the explicit penalty serves as a regularizer for the model to be more conservative about its reasoning. Such regularization boosts performance while requiring a slightly longer thinking process, which nevertheless remains shorter than the Deepseek-7B baseline.

对错误答案的显式惩罚可以正则化思考。 最后，为错误解答引入显式惩罚（GRPO+LEAD）取得了最高的 Pass@1。 在 AIME24 上，Pass@1 和 Cons@32 相较 GRPO+LAD 阶段分别提高 2.6%（$0.458\to 0.470$）和 4.3%（$0.767\to 0.800$）。 在 AIME25 上，Pass@1 也提高 6.2%（$0.325\to 0.345$），详见 表1。 值得注意的是，这些增益伴随着 AIME24 上平均解答长度适度增加，从约 5,300 个 token 增至 6,104 个 token。 图3 展示了这种权衡：低预算范围内会牺牲性能，但在 AIME25 上，当预算超过 5k token 时，GRPO+LEAD 仍优于 GRPO+LAD。 这些结果表明，显式惩罚充当了一种正则化，使模型对自己的推理更加审慎。 这种正则化提高了性能，代价是需要略长的思考过程，但其长度仍短于 Deepseek-7B 基线。

Overall, these ablation results confirm that all three enhancements---length-dependent accuracy, difficulty-aware advantage reweighting, and explicit penalties---collectively reduce verbosity, strengthen mathematical skills on harder questions, and elevate precision in final predictions.

总体而言，这些消融结果确认，长度依赖准确率、难度感知优势重加权和显式惩罚三项增强共同减少了冗长输出，强化了模型在困难问题上的数学能力，并提高了最终预测的精确率。

5.3 Comparison with Baselines

We next evaluate GRPO-LEAD at the 14B scale and compare it against two strong baselines under a 14k-token generation budget: DeepSeek-14B and the state-of-the-art Light-R1-14B-DS. Table 2 presents results on AIME24 and AIME25, including both our intermediate model (LEAD-stage1) and our final model (LEAD-stage2).

接下来，我们在 14B 规模上评估 GRPO-LEAD，并在 14k token 生成预算下与两个强基线比较：DeepSeek-14B 和当前最佳方法 Light-R1-14B-DS。 表2 给出了 AIME24 和 AIME25 上的结果，其中包括我们的中间模型（LEAD-stage1）和最终模型（LEAD-stage2）。

AIME24 Performance. LEAD-stage1 achieves a Cons@32 of 0.833, matching Light-R1-14B-DS and exceeding DeepSeek-14B by 4.1%. Its Pass@1 outperforms DeepSeek-14B by 2.4% and closely approaches Light-R1-14B-DS. Crucially, LEAD-stage1 produces more concise responses than both baselines, with more than 800 tokens less on average. Building on these gains, LEAD-stage2 pushes performance further, delivering the highest Cons@32 (4% above Light-R1-14B-DS) and the best Pass@1, while reducing average solution length to 8,267 tokens.

AIME24 性能。 LEAD-stage1 的 Cons@32 达到 0.833，与 Light-R1-14B-DS 持平，并比 DeepSeek-14B 高 4.1%。 它的 Pass@1 比 DeepSeek-14B 高 2.4%，并接近 Light-R1-14B-DS。 更重要的是，LEAD-stage1 生成的回答比两个基线都更简洁，平均少 800 多个 token。 在这些收益的基础上，LEAD-stage2 进一步提升性能，取得最高的 Cons@32（比 Light-R1-14B-DS 高 4%）和最佳 Pass@1，同时将平均解答长度降至 8,267 个 token。

AIME25 Performance. LEAD-stage1 yields a Cons@32 of 0.767, matching Light-R1-14B-DS and exceeding DeepSeek-14B by 21.2%. Its Pass@1 (0.523) outperforms DeepSeek-14B by 21.9% and Light-R1-14B-DS by 3.6%. Crucially, LEAD-stage1 produces more concise responses than both baselines, with its solutions averaging 9,371 tokens. Building on these gains, LEAD-stage2 pushes performance further, delivering the highest Cons@32 (matching Light-R1-14B-DS at 0.767) and the best Pass@1 (0.539), while reducing average solution length to 8,668 tokens.

AIME25 性能。 LEAD-stage1 的 Cons@32 达到 0.767，与 Light-R1-14B-DS 持平，并比 DeepSeek-14B 高 21.2%。 它的 Pass@1（0.523）比 DeepSeek-14B 高 21.9%，比 Light-R1-14B-DS 高 3.6%。 更重要的是，LEAD-stage1 生成的回答比两个基线都更简洁，其解答平均为 9,371 个 token。 在这些收益的基础上，LEAD-stage2 进一步提升性能，取得最高的 Cons@32（与 Light-R1-14B-DS 的 0.767 持平）和最佳 Pass@1（0.539），同时将平均解答长度降至 8,668 个 token。

Overall, both LEAD-stage1 and LEAD-stage2 deliver substantial improvements over DeepSeek-14B and Light-R1-14B-DS, simultaneously boosting correctness and conciseness under a constrained (14k-token) budget. Remarkably, training LEAD-stage1 for just 100 steps—requiring only about 24 hours on eight H20 GPUs—already matches Light-R1-14B-DS on Cons@32 and outperforms it on AIME25 Pass@1 while producing shorter solutions, underscoring the practical efficiency of GRPO-LEAD for large-scale math problem-solving.

总体而言，LEAD-stage1 和 LEAD-stage2 相较 DeepSeek-14B 与 Light-R1-14B-DS 均取得显著提升，在受限的 14k token 预算下同时提高了正确性与简洁性。 值得注意的是，LEAD-stage1 只训练 100 步，在 8 张 H20 GPU 上约需 24 小时，便已在 Cons@32 上追平 Light-R1-14B-DS，并在生成更短解答的同时取得更高的 AIME25 Pass@1，这凸显了 GRPO-LEAD 在大规模数学问题求解中的实际效率。

6. Conclusion

We introduced GRPO-LEAD, a reinforcement learning framework designed for mathematical reasoning tasks. By extending Group Relative Policy Optimization with three major components—(1) a length-dependent accuracy reward to discourage overly verbose solutions, (2) an explicit negative penalty that clarifies the boundary between correct and incorrect answers, and (3) a difficulty-aware advantage reweighting scheme to prioritize tougher problems—GRPO-LEAD addresses key challenges in structured problem-solving.

我们提出了 GRPO-LEAD，一种面向数学推理任务设计的强化学习框架。 GRPO-LEAD 通过为组相对策略优化扩展三个主要组件，解决结构化问题求解中的关键挑战：（1）使用长度依赖的准确率奖励抑制过度冗长的解答；（2）使用显式负惩罚明确正确答案与错误答案之间的边界；（3）使用难度感知的优势重加权方案优先处理更困难的问题。

Empirical evaluations on two AIME benchmarks show that GRPO-LEAD not only speeds up convergence but also strengthens the model’s reasoning capability while keeping solution paths concise. Our 14B-scale experiments further confirm that GRPO-LEAD achieves state-of-the-art performance by balancing output brevity with high problem-solving accuracy. Although open questions remain—particularly in managing partial correctness and extending these techniques to broader domains—our findings suggest that reward shaping and difficulty modeling are pivotal in developing more robust and aligned language models for complex mathematical reasoning.

两个 AIME 基准上的实证评估表明，GRPO-LEAD 不仅加快了收敛，还在保持解答路径简洁的同时增强了模型的推理能力。 我们的 14B 规模实验进一步确认，GRPO-LEAD 通过平衡输出简洁性与较高的问题求解准确率，取得了当前最佳性能。 尽管仍存在开放问题，尤其是如何处理部分正确性以及如何将这些技术扩展到更广泛的领域，但我们的发现表明，奖励塑形与难度建模对于开发更稳健、更符合预期的复杂数学推理语言模型至关重要。

Limitations

Although our techniques for encouraging concise solutions and difficulty-balanced learning may transfer to other domains, the gains reported here are specific to mathematical reasoning tasks. Further studies are needed to evaluate the effectiveness of GRPO-LEAD on broader question-answering or logical reasoning domains, where correctness signals and domain structures can differ substantially.

尽管我们用于鼓励简洁解答与难度平衡学习的技术可能迁移到其他领域，但本文报告的收益仅针对数学推理任务。 还需要进一步研究 GRPO-LEAD 在更广泛的问答或逻辑推理领域中的有效性，因为这些领域的正确性信号与领域结构可能存在显著差异。

Additionally, we only have access to a limited amount of compute, which prevents us from conducting more comprehensive experiments. For instance, we currently cannot provide the validation curve for the 7B model in the ablation study that adds an explicit penalty. This is due to an error in the validation code after upgrading to the newest VERL version, and we currently do not have the compute to reproduce it. A comparison with the original GRPO model is also missing, except for the curve shown in Figure 2, because the checkpoint was stored on a rented server that was automatically released as we were writing the paper. We also couldn't formally perform a hyperparameter search to showcase the rationale behind choosing the hyperparameters for our designed modifications.

此外，我们可用的计算资源有限，因此无法开展更全面的实验。 例如，在加入显式惩罚的消融研究中，我们目前无法提供 7B 模型的验证曲线。 这是因为升级到最新 VERL 版本后，验证代码出现错误，而我们目前没有足够算力复现实验。 除 图2 所示曲线外，我们也缺少与原始 GRPO 模型的比较，因为相应检查点存储在租用服务器上，而该服务器在我们撰写论文时已被自动释放。 我们也无法正式执行超参数搜索，以展示所设计改进中各超参数取值的依据。