---

ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning

LLMReasoningRLTMLR加州大学圣巴巴拉分校MIT-IBM Watson AI LabMIT CSAIL

Hou B, Zhang Y, Ji J, et al. ThinkPrune: pruning long chain-of-thought of LLMs via reinforcement learning[PP/OL]. V1. arXiv (2025-04-02)[2026-11-04]. DOI:10.48550/arXiv.2504.01296.

https://arxiv.org/abs/2504.01296v1

https://github.com/UCSB-NLP-Chang/ThinkPrune

---

ThinkPrune：通过强化学习剪枝大语言模型的长思维链

Abstract

We present ThinkPrune, a simple yet effective method for pruning the thinking length for long-thinking LLMs, which have been found to often produce inefficient and redundant thinking processes. Existing preliminary explorations of reducing thinking length primarily focus on forcing the thinking process to early exit, rather than adapting the LLM to optimize and consolidate the thinking process, and therefore the length-performance tradeoff observed so far is sub-optimal. To fill this gap, ThinkPrune offers a simple solution that continuously trains the long-thinking LLMs via reinforcement learning (RL) with an added token limit, beyond which any unfinished thoughts and answers will be discarded, resulting in a zero reward. To further preserve model performance, we introduce an iterative length pruning approach, where multiple rounds of RL are conducted, each with an increasingly more stringent token limit.

我们提出 ThinkPrune，一种简单而有效的方法，用于剪枝长思考大语言模型的思考长度；已有研究发现，这类模型经常产生低效且冗余的思考过程。 现有减少思考长度的初步探索，主要着眼于强制思考过程提前退出，而非让大语言模型通过适应性训练来优化和精简思考过程，因此迄今观察到的长度与性能权衡并非最优。 为填补这一空白，ThinkPrune 提供了一种简单方案：通过强化学习（RL）持续训练长思考大语言模型，并加入 token 数量上限，超过上限时，任何未完成的思考和答案都会被丢弃，从而得到零奖励。 为进一步保持模型性能，我们引入迭代式长度剪枝方法，进行多轮强化学习，并在每一轮施加更严格的 token 上限。

We observed that ThinkPrune results in a remarkable performance-length tradeoff — on the AIME24 dataset, the reasoning length of DeepSeek-R1-Distill-Qwen-1.5B can be reduced by half with only 2% drop in performance. We also observed that after pruning, the LLMs can bypass unnecessary steps while keeping the core reasoning process complete. Code is available at https://github.com/UCSB-NLP-Chang/ThinkPrune.

我们观察到，ThinkPrune 实现了出色的性能与长度权衡：在 AIME24 数据集上，DeepSeek-R1-Distill-Qwen-1.5B 的推理长度可以缩短一半，而性能仅下降 2%。 我们还观察到，剪枝后的大语言模型能够跳过不必要的步骤，同时保持核心推理过程的完整。 代码地址为 https://github.com/UCSB-NLP-Chang/ThinkPrune。

1. Introduction

Recent advances in large language models (LLMs) have demonstrated the effectiveness of inference-time scaling through reinforcement learning, where LLMs learn to produce long and sophisticated reasoning behaviors such as self-refection and verification, significantly increasing their performance on a wide range of benchmarks. However, one key challenge of inference-time scaling is the significant number of tokens produced at inference time, leading to high computational and memory overhead. For example, on the MATH500 benchmark, the DeepSeek-R1-Distill-Qwen-1.5B model generates solutions with more than 15,000 tokens on average, while many of the questions could have been solved with fewer than 1,000 tokens by regular LLMs. This highlights the issue of over-thinking, where many reasoning steps might be redundant or inefficient.

大语言模型（LLM）的近期进展表明，通过强化学习实现推理时扩展是有效的：大语言模型学会生成较长且复杂的推理行为，例如自我反思和验证，从而显著提高其在各类基准上的性能。 然而，推理时扩展的一项关键挑战是推理阶段会生成大量 token，带来较高的计算和内存开销。 例如，在 MATH500 基准上，DeepSeek-R1-Distill-Qwen-1.5B 模型生成的解答平均超过 15,000 个 token，而其中许多问题本可以由常规大语言模型用不到 1,000 个 token 解决。 这凸显了 过度思考 问题：许多推理步骤可能是冗余或低效的。

There are now some preliminary explorations of limiting the generation length via budget-forcing, where, when reaching a given token limit, the thinking process is forced to early exit, e.g., by appending an end-of-thinking token and producing an answer right away. Figure 1(a) (left) shows an example output of S1, a budget-forcing method, under a low budget (2000 thinking tokens), where the thinking process spent 453 tokens just to understand the problem, and therefore could not complete its second round of thought within the budget. Accordingly, a non-trivial performance drop would occur as the budget gets tighter, as shown in Figure 1(b) (blue lines).

目前已有一些通过 预算强制 限制生成长度的初步探索，即在达到给定 token 上限时，强制思考过程提前退出，例如追加一个结束思考的 token，并立即生成答案。 图 1（a）左侧展示了预算强制方法 S1 在较低预算（2,000 个思考 token）下的输出示例：其思考过程仅理解题意就花费了 453 个 token，因此未能在预算内完成第二轮思考。 相应地，随着预算收紧，性能会出现明显下降，如图 1（b）中的蓝线所示。

(a) Example Reasoning Chains.(b) Accuracy vs. Think Length.

预算强制与 ThinkPrune 的比较。（a）推理链示例：在 2,000 个思考 token 的预算下，对原始模型应用预算强制，会在识别错误之前耗尽全部 token 预算，从而给出错误答案。相比之下，经过 ThinkPrune 训练的模型能够更高效地解题，使用更少的 token 并给出正确答案。（b）原始大语言模型与经过 ThinkPrune 训练的模型，在不同思考 token 预算下的性能。

图1：预算强制与 ThinkPrune 的比较。（a）推理链示例：在 2,000 个思考 token 的预算下，对原始模型应用预算强制，会在识别错误之前耗尽全部 token 预算，从而给出错误答案。相比之下，经过 ThinkPrune 训练的模型能够更高效地解题，使用更少的 token 并给出正确答案。（b）原始大语言模型与经过 ThinkPrune 训练的模型，在不同思考 token 预算下的性能。

However, this is apparently sub-optimal. When the budget is low, a much more sensible solution is to remove the redundant thinking or unimportant steps (such as the problem paraphrasing), rather than maintaining the inefficient thinking and getting it killed. As a result, the performance drop as thinking length shrinks may have been significantly overestimated. So far, there have not been explorations that seek to fine-tune the model to adapt to smaller thinking budgets, and thus, we still do not have a good estimate of how much redundancy there is in the long thinking process that can be removed. Specifically, the following research questions remain unanswered:

然而，这显然并非最优做法。 当预算较低时，更合理的方案是去除冗余思考或不重要的步骤，例如复述题意，而不是保留低效的思考过程，再将其强行截断。 因此，思考长度缩短所导致的性能下降，可能被显著高估了。 迄今尚未有研究尝试微调模型，使其适应更小的思考预算，因此我们仍无法准确估计，长思考过程中究竟有多少冗余可以去除。 具体而言，以下研究问题仍未得到回答：

• Can we fine-tune an LLM with long CoT to prune its thinking length, while minimizing the performance drop?
• What would be the length-performance tradeoff when the long CoT is pruned?
• What happens to the reasoning chain when it gets pruned? What steps or words are most likely to be pruned?

• 能否通过微调具有长思维链的大语言模型，剪短其思考长度，同时尽量减小性能下降？
• 对长思维链进行剪枝后，长度与性能之间会呈现怎样的权衡？
• 推理链被剪枝后会发生什么变化？哪些步骤或词语最有可能被剪除？

In this paper, we seek to fill this gap. We propose ThinkPrune, a simple yet effective length-pruning strategy for LLMs with long CoT, which enforces a maximum generation length during RL training. Specifically, given an LLM with long CoT, we perform continuous RL following the same scheme of DeepSeek-R1, except that we impose a strict token limit during training (e.g., 4,000 tokens for both reasoning and answer tokens). Any tokens beyond this limit are discarded before reward computation. This means that even if the model generates a correct answer beyond the allowed length, it still receives a reward of 0 because the answer is clipped and cannot be extracted. The task performance can be further preserved with an iterative pruning strategy, where multiple rounds of RL are conducted with increasingly more stringent length limits.

本文旨在填补这一空白。 我们提出 ThinkPrune，一种面向长思维链大语言模型、简单而有效的长度剪枝策略，在强化学习训练期间强制设置最大生成长度。 具体而言，给定一个具有长思维链的大语言模型，我们遵循与 DeepSeek-R1 相同的方案持续进行强化学习，区别在于训练时施加严格的 token 上限，例如推理与答案合计最多 4,000 个 token。 超过这一上限的所有 token，都会在计算奖励之前被丢弃。 这意味着，即便模型在允许的长度之外生成了正确答案，它仍会得到 0 奖励，因为答案已被截断，无法提取。 还可以通过迭代式剪枝策略进一步保持任务性能，即进行多轮强化学习，并逐轮收紧长度限制。

Extensive evaluation demonstrates a strong trade-off between generation length and performance for ThinkPrune. For example, ThinkPrune reduces the average generation length from 10,355 to 3,574 tokens for the DeepSeek-R1-Distill-Qwen-1.5B model, while also improving average accuracy across four math benchmarks. Although there is a slight performance drop of 2% for the DeepScaleR-1.5B-Preview and QwQ-32B models, our method achieves a comparable reduction in generation length — from 5,914 to 3,370 tokens, and from 8,763 to 4494, respectively. Figure 1(b) (yellow lines) also shows the improved accuracy-thinking length tradeoff. Further analysis shows that ThinkPrune helps LLMs avoid unnecessary reasoning steps while maintaining focus on solving the question, as shown in Figure 1(a) (right), where problem understanding drops to 166 tokens. These findings provide valuable insights into improving the inference-time reasoning efficiency for long-COT LLMs.

大量评估表明，ThinkPrune 能够在生成长度与性能之间取得良好权衡。 例如，对于 DeepSeek-R1-Distill-Qwen-1.5B 模型，ThinkPrune 将平均生成长度从 10,355 个 token 减少到 3,574 个，同时还提高了四个数学基准上的平均准确率。 对于 DeepScaleR-1.5B-Preview 和 QwQ-32B 模型，尽管性能略降约 2%，我们的方法仍实现了相近程度的生成长度缩减，分别从 5,914 减至 3,370 个 token，以及从 8,763 减至 4,494 个 token。 图 1（b）中的黄线也展示了准确率与思考长度权衡的改善。 进一步分析表明，ThinkPrune 能帮助大语言模型避免不必要的推理步骤，同时专注于解题；如图 1（a）右侧所示，理解题意所用的 token 数降至 166。 这些发现为提高长思维链大语言模型在推理阶段的效率提供了有价值的认识。

2. Related Work

2.1. Reinforcement Learning for LLM Reasoning

Reinforcement learning (RL) has shown strong potential in improving the reasoning abilities of LLMs across various domains, such as math and coding, and complicated browser surfing. The resulting long-COT LLMs such as OpenAI-o3 and DeepSeek-R1 significantly outperform short-COT LLMS and demonstrate that reinforcement learning with verifiable reward (RLVR) can encourage LLMs to develop deep thinking behaviors, such as broad exploration and feasibility checks, without relying on complex reasoning data generation methods like Monte-Carlo Tree Search. However, these behaviors often lead to much longer reasoning traces, sometimes several times longer than those produced by short COT LLMs, creating an “overthinking” issue that largely increases inference costs. Several recent works have shown that this extended reasoning often includes redundant or unnecessary verification and reflection, even on simple problems. Our work follows the standard RL training pipeline without changing the reward function and shows that it is possible to retain strong reasoning performance while significantly reducing overthinking.

强化学习（RL）在提高大语言模型的推理能力方面展现出强大潜力，涉及数学、编程以及复杂的浏览器操作等多个领域。 由此得到的 OpenAI-o3、DeepSeek-R1 等长思维链大语言模型，显著优于短思维链大语言模型，并表明使用可验证奖励的强化学习（RLVR）能够促使大语言模型形成深入思考行为，例如广泛探索与可行性检查，而无需依赖蒙特卡洛树搜索等复杂的推理数据生成方法。 然而，这些行为往往会产生更长的推理轨迹，有时达到短思维链大语言模型的数倍，形成大幅增加推理成本的“过度思考”问题。 近期一些工作表明，即便面对简单问题，这种延长的推理也常常包含冗余或不必要的验证与反思。 我们的工作遵循标准强化学习训练流程，不改变奖励函数，并表明在大幅减少过度思考的同时，仍然能够保持较强的推理性能。

2.2. Efficient Long Chain-of-Thought LLM

Several works have aimed to improve token efficiency in long chain-of-thought (CoT) LLMs. For example, KimiTeam et al., Chen et al., and Shen et al. reduce the length of reasoning by adding a length penalty during RL training. Other works, such as Hao et al. and Geiping et al., represent reasoning as an optimization over latent vectors instead of text tokens, which helps shorten the reasoning process. While these methods are effective when training long CoT LLMs from short COT ones, there have been few works on reducing the reasoning length for trained LLMs.

已有多项工作致力于提高长思维链（CoT）大语言模型的 token 效率。 例如，KimiTeam 等人、Chen 等人和 Shen 等人通过在强化学习训练中加入长度惩罚，缩短推理长度。 Hao 等人和 Geiping 等人的其他工作，则将推理表示为对潜在向量而非文本 token 的优化，从而帮助缩短推理过程。 虽然这些方法在将短思维链模型训练为长思维链模型时是有效的，但针对已经训练好的大语言模型缩短推理长度的工作仍然很少。

The most relevant works to ours are test-time methods that shorten reasoning with early exit strategies. These methods add stopping tokens or limit the maximum number of generated tokens during testing. However, they often lead to significant performance drops, especially when early stopping occurs too soon in the reasoning process. In contrast, our work presents a simple and effective method with further RL training to reduce reasoning length without sacrificing performance.

与我们最相关的是通过提前退出策略缩短推理过程的测试时方法。 这些方法会在测试期间加入停止 token，或限制生成 token 的最大数量。 然而，它们往往会导致明显的性能下降，尤其是在推理过程过早停止时。 相比之下，我们提出了一种简单而有效的方法，通过进一步强化学习训练，在不牺牲性能的前提下减少推理长度。

3. Method

3.1. Overview

Denote ${\mathcal{M}}_{{\mathit{\theta }}_0}(\cdot )$ as an LLM capable of performing long CoT. Given a query $\mathit{q}$, a sample answer, along with a long reasoning chain, is sampled from the LLM's output distribution, i.e., $\mathit{Y}\sim {\mathcal{M}}_{{\mathit{\theta }}_0}(\mathit{q})$. Our goal is to fine-tune the LLM, such that its output length is reduced while the overall performance is desirably maintained.

将能够进行长思维链推理的大语言模型记作 ${\mathcal{M}}_{{\mathit{\theta }}_0}(\cdot )$。 给定问题 $\mathit{q}$，从大语言模型的输出分布中采样一个答案及其长推理链，即 $\mathit{Y}\sim {\mathcal{M}}_{{\mathit{\theta }}_0}(\mathit{q})$。 我们的目标是微调这一大语言模型，使其输出长度缩短，同时尽可能保持整体性能。

ThinkPrune is a simple yet effective RL strategy to achieve the length reduction goal. In what follows, we will first introduce the RL framework of ThinkPrune, and then introduce an iterative length reduction strategy to better maintain the performance.

ThinkPrune 是一种简单而有效的强化学习策略，用于实现缩短长度的目标。 下文将首先介绍 ThinkPrune 的强化学习框架，然后介绍一种迭代式长度缩减策略，以更好地保持性能。

3.2. Reinforcement Learning with Length Clipping

ThinkPrune adopts a similar RL scheme to the DeepSeek-R1 model while reducing the generation length. Specifically, we adopt the group relative policy optimization (GRPO) algorithm. The reward function is almost the same as the DeepSeek-R1 framework, except that a length clipping is added. Formally, denote $L$ as the length limit. The reward function is defined as follows:

ThinkPrune 采用与 DeepSeek-R1 模型类似的强化学习方案，同时减少生成长度。 具体而言，我们采用组相对策略优化（GRPO）算法。 奖励函数与 DeepSeek-R1 框架几乎相同，唯一的区别 是加入了 长度截断。 形式化地，将长度上限记为 $L$。 奖励函数定义如下：

$$\begin{array}{}\text{(1)}& \begin{array}{rl}& R(\mathit{Y},\mathit{q};L)=\\ & \{\begin{array}{ll}1& \begin{array}{rl}& \text{if an answer can be extracted}\\ & \text{from }\mathrm{clip}(\mathit{Y},L)\\ & \text{and is correct,}\end{array}\\ 0& \text{otherwise;}\end{array}\end{array}\end{array}$$

where clip$(\mathit{Y};L)$ represents clipping the output $\mathit{Y}$ to length $L$. In other words, the only difference from the DeepSeek-R1 reward is that the model-generated output is clipped to $L$ before the reward is evaluated. In this way, output with length above $L$ would not be able to produce a valid answer since it is cut off, thus receiving a zero reward. This clipping operation effectively encourages the model to produce answers below the length limit. Such reward design is very simple, so it does not involve any hyperparameter tuning or reward engineering, and can ensure training stability. Also, since it involves minimal tweaks to DeepSeek-R1's training strategy, the proposed solution helps to maintain the long reasoning capabilities inherent in the base model.

其中，clip$(\mathit{Y};L)$ 表示将输出 $\mathit{Y}$ 截断至长度 $L$。 换言之，与 DeepSeek-R1 奖励的唯一区别在于，在评估奖励之前，先将模型生成的输出截断至 $L$。 这样一来，长度超过 $L$ 的输出由于被截断而无法给出有效答案，因此获得零奖励。 这一截断操作能够有效鼓励模型在长度上限以内生成答案。 这种奖励设计非常简单，不涉及超参数调优或奖励工程，并能保证训练稳定性。 此外，由于对 DeepSeek-R1 训练策略的改动很小，所提出的方案有助于保留基础模型原有的长推理能力。

During training, we also append a simple system prompt into each training example to explicitly tell the model the length limit, such as The output of the assistant should be within L tokens, to explicitly tell the model the length limit. The full system prompt for each model is shown in Appendix A.1.

训练期间，我们还会为每个训练样本追加一个简单的系统提示，以明确告知模型长度限制，例如 助手的输出应在 L 个 token 以内，从而显式传达长度上限。 各模型的完整系统提示见附录 A.1。

3.3. Iterative Length Pruning Strategy

The success of the proposed algorithm relies on the choice of the length limit, $L$ — if $L$ is set too stringently compared to the original output length of the base model, then the task performance can be seriously compromised.

所提出算法的成功依赖于长度上限 $L$ 的选择：如果相对于基础模型的原始输出长度，将 $L$ 设得过于严格，任务性能就可能受到严重损害。

Drawing inspiration from the iterative pruning strategy for reducing model parameters, we propose an iterative length pruning scheme. Denote $L^{\ast }$ as the target length constraint, we introduce a length schedule, $L_1>L_2>\cdots >L^{\ast }$. At each iteration $t$, we reduce the length constraint to $L_t$, and further fine-tune the model ${\mathit{\theta }}_{t-1}$ from last iteration to ${\mathit{\theta }}_t$, using the RL procedure described in Section 3.2. Such an iterative pruning strategy ensures that the LLM learns to recover the performance by gradually compacting its output reasoning chain.

受用于减少模型参数的迭代式剪枝策略启发，我们提出一种迭代式长度剪枝方案。 将目标长度约束记作 $L^{\ast }$，并引入长度调度序列 $L_1>L_2>\cdots >L^{\ast }$。 在每轮迭代 $t$ 中，我们将长度约束减小至 $L_t$，并使用第 3.2 节介绍的强化学习过程，将上一轮模型 ${\mathit{\theta }}_{t-1}$ 进一步微调为 ${\mathit{\theta }}_t$。 这种迭代式剪枝策略确保大语言模型能够通过逐渐压缩输出推理链，学会恢复性能。

A critical design choice of the iterative length pruning is the stopping criterion for each iteration of the RL training. In our implementation, we utilize AIME22 and AIME23 as the dataset as a validation set to choose the best checkpoint for the next RL iteration. To better balance model performance and generation length reduction, we allow up to a relative 10% drop in pass@1 accuracy on AIME-22 and AIME-23 compared to the original model. Among the checkpoints that meet this criterion, we select the one with the shortest average output length as the starting checkpoint.

迭代式长度剪枝中的一项关键设计，是每轮强化学习训练的停止准则。 在我们的实现中，使用 AIME22 和 AIME23 作为验证集，为下一轮强化学习选择最佳检查点。 为更好地平衡模型性能与生成长度的缩减，我们允许模型在 AIME-22 和 AIME-23 上的 pass@1 准确率，相对于原始模型最多下降 10%。 在满足这一条件的检查点中，我们选择平均输出长度最短的一个作为起始检查点。

4. Experiment

In this section, we conduct empirical evaluations to assess the effectiveness of our proposed method. We first present the experiment setup in Section 4.1. Then, we present the experiment results in Section 4.2.

本节通过实证评估，考察所提出方法的有效性。 首先在第 4.1 节介绍实验设置。 随后在第 4.2 节给出实验结果。

4.1. Experiment Setup

Backbone models. Representative open-sourced long reasoning LLMs include DeepSeek-R1 and QwQ, along with their distilled variants. In our experiments, we use three models from these families: Distill-Qwen-1.5B, DeepScaleR-1.5B-Preview, and QwQ-32B.

骨干模型。 具有代表性的开源长推理大语言模型包括 DeepSeek-R1、QwQ 及其蒸馏变体。 实验使用这些模型家族中的三个模型：Distill-Qwen-1.5B、DeepScaleR-1.5B-Preview 和 QwQ-32B。

We group these models into two categories based on the extent of their training: unsaturated and saturated models. Specifically, while Deepseek-R1-Distill models have been widely used, these models are directly fine-tuned on the output of DeepSeek-R1 in a supervised fine-tuning manner instead of RL, which could limits their full potential. Previous work, such as Luo et al., has shown that further training these models with RL can improve their performance, especially on math benchmarks. Based on this, we treat Distill-Qwen-1.5B as an unsaturated model and DeepScaleR-1.5B-Preview as its saturated version, since it is further trained with RL. Similarly, QwQ-32B is also trained with RL and is considered a saturated model. This selection of models, covering both unsaturated and saturated types, allows us to more thoroughly evaluate the effects of thinking length pruning.

根据训练充分程度，我们将这些模型划分为两类：未饱和 模型和 饱和 模型。 具体而言，虽然 Deepseek-R1-Distill 模型已被广泛使用，但它们只是直接对 DeepSeek-R1 的输出进行有监督微调，而非强化学习，这可能限制了其潜力的充分发挥。 Luo 等人的先前工作表明，进一步使用强化学习训练这些模型，可以改善其性能，尤其是在数学基准上。 基于这一点，我们将 Distill-Qwen-1.5B 视为未饱和模型，将经过进一步强化学习训练的 DeepScaleR-1.5B-Preview 视为它的饱和版本。 同样，QwQ-32B 也经过强化学习训练，因此被视为饱和模型。 这一模型选择同时覆盖未饱和与饱和两类，使我们能够更全面地评估思考长度剪枝的影响。

Training datasets. Previous work has shown that even a small but high-quality training dataset can improve the LLM performance via RL. Therefore, we utilize a small number of data for model training by only using the historical AIME and AMC math questions. We use the preprocessed data from Prime and take the AIME-AMC subset for training. In total, the training dataset consists of 2470 distinct training examples.

训练数据集。 先前工作表明，即便规模很小，只要训练数据质量足够高，也能通过强化学习提高大语言模型性能。 因此，我们仅使用历年的 AIME 和 AMC 数学题，以少量数据进行模型训练。 我们采用 Prime 预处理后的数据，并选取其中的 AIME-AMC 子集进行训练。 训练数据集共包含 2,470 个不同的训练样本。

Comparisons. We compare with the original backbone models without pruning. Additionally, we include the budget-forcing method for length reduction, which enforces a maximum number of thinking tokens by appending the end-of-thinking token delimiter (detailed implementation is described in Appendix A.1). For our method, we report the following two variants:

比较方法。 我们与未经剪枝的原始骨干模型进行比较。 此外，还纳入预算强制方法进行长度缩减；该方法通过追加结束思考的 token 分隔符，强制限制思考 token 的最大数量，详细实现见附录 A.1。 对于我们的方法，报告以下两种变体：

• One-shot length pruning: We set the maximum length to 4,000/3,000/2,000 tokens respectively and directly perform RL to reduce the generation length of the LLM.
• Iterative length pruning: Starting from a higher generation length budget, we perform multi-round RL training and gradually decrease the maximum length after each iteration.

• 单次长度剪枝：分别将最大长度设为 4,000、3,000、2,000 个 token，直接通过强化学习缩短大语言模型的生成长度。
• 迭代式长度剪枝：从较高的生成长度预算开始，进行多轮强化学习训练，并在每轮迭代后逐步降低最大长度。

Implementation details. We use the Verl RL framework for high-performance training. All models are trained with a batch size of 128. The number of rollouts for each question is set to 16 following prior works. We use the GROP algorithm. For both one-shot and iterative pruning, we employ the same checkpoint selection strategy as mentioned in Section 3.3, where the checkpoint with the shortest average output length while maintaining a relative 10% accuracy drop is selected.

实现细节。 我们使用 Verl 强化学习框架进行高性能训练。 所有模型的训练批大小均为 128。 遵循先前工作，每个问题的 rollout 数设为 16。 我们使用 GRPO 算法。 对于单次剪枝和迭代式剪枝，均采用第 3.3 节介绍的相同检查点选择策略：在满足准确率相对下降不超过 10% 的条件下，选择平均输出长度最短的检查点。

Evaluation configurations. We follow prior works to include the following evaluation datasets: MATH-500, AIME24, AMC23, and OlympiadBench. We use the versions of these dataset hosted in the Qwen2.5-Math GitHub repository for ease of reference. Following DeepSeek R1, we set the maximum generation length (including both the thinking tokens and answer tokens) to 32,768 tokens for all the models, far above the maximum token length during our training. For each testing question, we sample $N$ outputs with a temperature of 0.6 and a top-p value of 0.95, and we report the average accuracy of these $N$ outputs.

评估配置。 遵循先前工作，我们采用以下评估数据集：MATH-500、AIME24、AMC23 和 OlympiadBench。 为便于对照，我们使用 Qwen2.5-Math GitHub 仓库提供的数据集版本。 遵循 DeepSeek R1，我们将所有模型的最大生成长度设为 32,768 个 token，包括思考 token 和答案 token，远高于训练期间的最大 token 长度。 对于每道测试题，以温度 0.6 和 top-p 0.95 采样 $N$ 个输出，并报告这 $N$ 个输出的平均准确率。

The number of samplings varies depending on the model size and dataset size. Specifically, for the two 1.5B models, we use $N=64$ for AIME24 and AMC23 and $N=16$ for MATH500 and OlympiadBench. This also refers to the evaluation hyper-parameters of DeepSeek R1, where the number of sampled responses are adjusted between 4 and 64 depending on the test set size to balance the variance and evaluation cost. For QwQ-32B, we sample $N=16$ responses for each question, given its large size and high computational cost.

采样次数会根据模型规模和数据集大小而变化。 具体而言，对于两个 1.5B 模型，在 AIME24 和 AMC23 上使用 $N=64$，在 MATH500 和 OlympiadBench 上使用 $N=16$。 这一设置也参考了 DeepSeek R1 的评估超参数：根据测试集大小，将回答采样数在 4 到 64 之间调整，以平衡方差和评估成本。 对于 QwQ-32B，考虑到其规模较大、计算成本较高，每个问题采样 $N=16$ 个回答。

Since the accuracy evaluation requires complex evaluation of mathematical expressions, we adopt the math evaluator from Qwen-2.5-math, which provides robust answer extraction and advanced expression comparison.

由于准确率评估需要对数学表达式进行复杂判断，我们采用 Qwen-2.5-math 的数学评估器，它能够稳健地提取答案，并进行高级表达式比较。

表1：ThinkPrune 的性能展示。准确率通过从大语言模型中采样多个回答并取平均来计算，以降低方差。

	Accuracy					Generation Length
	MATH 500	AIME	AMC	Olympiad Bench	Avg.	MATH 500	AIME	AMC	Olympiad Bench	Avg.
DeepSeek-R1-Distill-Qwen-1.5B
Original Model	82.9	29.4	70.3	44.7	56.8	5560	15484	10030	11526	10355
ThinkPrune-4k	83.8	29.0	73.6	46.5	58.2	2709	8301	4388	5529	5232
ThinkPrune-3k	83.7	27.8	71.8	44.9	57.1	2557	7968	4096	5140	4940
ThinkPrune-2k	82.9	27.0	72.2	45.6	56.9	2356	7574	3755	4913	4650
ThinkPrune-4k→3k	83.9	26.9	71.4	46.0	57.1	2209	6389	3422	4229	4062
ThinkPrune-4k→3k→2k	83.2	27.1	73.2	46.2	57.4	1938	5631	3039	3687	3574
DeepScaleR-1.5B-Preview
Original Model	88.5	40.3	81.2	52.7	65.7	3084	9463	5202	5907	5914
ThinkPrune-4k	87.1	37.2	80.2	51.4	64.0	2212	6366	3516	4055	4037
ThinkPrune-3k	86.5	34.3	78.8	50.6	62.6	1991	5809	3122	3583	3626
ThinkPrune-2k	86.1	33.3	77.7	49.7	61.7	1880	5528	2961	3348	3429
ThinkPrune-4k→3k	87.1	38.4	79.9	51.6	64.2	2086	5869	3278	3731	3741
ThinkPrune-4k→3k→2k	86.9	36.5	79.4	50.1	63.2	1881	5301	2963	3334	3370
QwQ-32B
Original Model	95.1	78.8	97.5	71.1	85.6	4289	13822	7442	9497	8763
ThinkPrune-4k	94.0	76.3	95.8	68.2	83.5	2552	8787	4173	5687	5300
ThinkPrune-3k	94.0	75.0	95.8	68.6	83.3	2341	8308	3943	5413	5001
ThinkPrune-2k	93.5	73.3	95.5	68.6	82.7	2133	8232	3770	5160	4824
ThinkPrune-4k→3k	94.0	73.8	96.1	67.6	82.9	2308	8176	3959	5301	4936
ThinkPrune-4k→3k→2k	93.8	72.5	95.9	67.3	82.4	2162	7631	3441	4742	4494

ThinkPrune 的性能展示。准确率通过从大语言模型中采样多个回答并取平均来计算，以降低方差。

4.2. Main Experiment: Length Pruning

One-shot length pruning enables significant thinking length reduction. We first evaluate the effectiveness of one-shot pruning in Table 1 (top 3 rows in each section), where the LLMs are directly trained with a length limit 2k/3k/4k. We highlight the following observations. First, the simple strategy can effectively reduce the generation length with moderate negative effect on the performance. For the unsaturated model, DeepSeek-R1-Distill-Qwen-1.5B, the length reduction rate can be up to 50% with one-shot length pruning. At the meantime, the average performance is well-maintained and even slightly improved.

单次长度剪枝能够显著缩短思考长度。 我们首先在表 1中评估单次剪枝的有效性，对应各分组中的前三条剪枝结果；这些大语言模型直接在 2k、3k、4k 的长度上限下训练。 我们重点指出以下观察结果。 首先，这种简单策略能够有效减少生成长度，而对性能的负面影响较为有限。 对于未饱和模型 DeepSeek-R1-Distill-Qwen-1.5B，单次长度剪枝可将长度缩短多达 50%。 与此同时，平均性能得到了良好保持，甚至略有提升。

For saturated models like DeepScaleR-1.5B-Preview and QwQ-32B, we observe a 40–50% reduction in token usage, with moderate performance degradation. This highlights the promising efficiency gains through length pruning, especially when models are initially over-generating. Second, we observe a consistent tradeoff between length and performance for all models under one-shot pruning. As we lower the token limit from 4k to 2k, the number of tokens goes down and the accuracy drops slightly. This suggests that cutting more reasoning tokens aggressively may also limit the reasoning capabilities of the LLMs.

对于 DeepScaleR-1.5B-Preview 和 QwQ-32B 等饱和模型，我们观察到 token 用量减少了 40%–50%，伴随一定程度的性能下降。 这凸显了长度剪枝带来的可观效率收益，尤其是在模型原本存在过量生成时。 其次，对于所有模型，单次剪枝都呈现出一致的长度与性能权衡。 随着 token 上限从 4k 降至 2k，token 数量减少，准确率也略有下降。 这说明，过于激进地剪除更多推理 token，也可能限制大语言模型的推理能力。

Another interesting phenomenon is that even though the model is trained with an explicit length limit, it often goes beyond that limit at test time. Particularly, we observe that although trained with an explicit length limit, the model can still generate long responses when the problem becomes more difficult such as on the AIME dataset. This shows that our length pruning does not hurt the model's deep thinking behavior, and it maintains the ability to perform complex reasoning for difficult questions.

另一个有趣的现象是，尽管训练时设置了明确的长度限制，模型在测试时仍经常超过这一上限。 特别是，我们观察到，即便使用明确的长度限制训练，当问题更加困难时，例如面对 AIME 数据集，模型仍能生成较长回答。 这表明，我们的长度剪枝没有损害模型的深度思考行为，它仍然保留了针对困难问题进行复杂推理的能力。

Iterative length pruning benefits performance preservation and length reduction. The second question we aim to explore is whether iterative length pruning can better main the original LLM performance with a decent thinking length pruning compared to one-shot pruning. As shown in Table 1 (last two rows in each section), we start with LLMs trained using a 4k length limit and then iteratively apply RL with reduced length limits from 3k to 2k. Our findings are as follows. First, for the two 1.5B models, iterative pruning leads to better performance with either shorter or similar response lengths compared to one-shot pruning under the same final length limit. For example, on DeepScaleR, ThinkPrune-4k→3k→2k outperforms the one-shot 2k model by 1.5% in accuracy while using 59 fewer tokens on average. Similarly, pruning from 4k to 3k leads to better results than directly pruning to 3k.

迭代式长度剪枝有助于保持性能并缩短长度。 我们希望探索的第二个问题是：相比单次剪枝，迭代式长度剪枝能否在有效剪短思考长度的同时，更好地保持大语言模型的原始性能。 如表 1各分组的最后两行所示，我们从已在 4k 长度上限下训练的模型出发，依次使用降至 3k、再降至 2k 的长度上限进行强化学习。 我们的发现如下。 首先，对于两个 1.5B 模型，在相同的最终长度上限下，迭代式剪枝能够以更短或相近的回答长度，取得优于单次剪枝的性能。 例如，在 DeepScaleR 上，ThinkPrune-4k→3k→2k 的准确率比单次 2k 剪枝模型高出 1.5%，同时平均少用 59 个 token。 同样，从 4k 剪枝至 3k，也优于直接剪枝至 3k。

Second, for the QwQ-32B model, performance drops slightly after length pruning, with an average decrease of 2.7%. Unlike the 1.5B models, iterative pruning does not help recover the lost performance. We believe this is because it struggles to maintain performance when the generation token budget is too tight. We will discuss this in more detail in the next paragraph.

其次，对于 QwQ-32B 模型，长度剪枝后的性能略有下降，平均降幅为 2.7%。 与 1.5B 模型不同，迭代式剪枝未能帮助恢复损失的性能。 我们认为，这是因为当生成 token 预算过于紧张时，该模型难以保持性能。 下一段将对此进行更详细的讨论。

在 AIME24 数据集上评估的长度剪枝训练过程中的性能与长度权衡。在训练早期，生成长度迅速下降，而性能仅有轻微降低。

图2：在 AIME24 数据集上评估的长度剪枝训练过程中的性能与长度权衡。在训练早期，生成长度迅速下降，而性能仅有轻微降低。

Performance-length trade-off along the pruning process. We observe that the experiment results in Table 1 indicates a positive correlation between the reasoning length and model performance on different benchmarks. To better understand the trade-off between reasoning length and model performance, we visualize how these two change during the length pruning training process on the AIME24 dataset (see Figure 2). During training, we evaluate the model every 20 steps and record both the performance and the average output length. We then identify the most effective checkpoints (the ones at the frontier of performance-length trade-off), defined as the checkpoints that achieve the best performance among all the checkpoints whose average output length is shorter than themselves. This gives us a clear picture of the best performance the model can reach at each length range, helping us visualize the efficiency boundary of the pruning process.

剪枝过程中的性能与长度权衡。 我们观察到，表 1中的实验结果表明，在不同基准上，推理长度与模型性能呈正相关。 为更好地理解推理长度与模型性能之间的权衡，我们将在 AIME24 数据集上进行长度剪枝训练时，这两个量的变化过程可视化，见图 2。 训练期间，每隔 20 步评估一次模型，并记录性能与平均输出长度。 随后，我们找出最有效的检查点，即性能与长度权衡前沿上的检查点：其性能优于所有平均输出长度比它更短的检查点。 这样便能清晰地看到模型在各个长度区间可以达到的最佳性能，从而将剪枝过程的效率边界可视化。

We highlight the following observations: First, mild length pruning leads to strong efficiency gains with minimal performance drop. When reasoning length reduction is moderate, there exists a favorable trade-off: significant token reductions can be achieved with only minor drops in performance. For example, we can reduce the average generation length of QwQ-32B from 14K to 8K while maintaining its performance close to the original model.

我们重点指出以下观察结果：首先，温和的长度剪枝能够带来显著的效率收益，而性能仅有极小下降。 当推理长度的缩减程度适中时，存在一种有利的权衡：仅需承受轻微的性能下降，就能显著减少 token 数量。 例如，我们可以将 QwQ-32B 的平均生成长度从 14K 缩短至 8K，同时将其性能保持在接近原始模型的水平。

Second, for the QwQ-32B model, we observe a distinct critical threshold (marked by the vertical dotted line in Figure 2). Beyond this threshold, further reducing the reasoning length causes a sharp and significant drop in performance. We also find enforcing 4k length limit leads to even worse results than 3k length limit on QwQ-32B. This drop partially explains why iterative pruning performs worse on QwQ-32B than on smaller models like 1.5B. While optimizing hyperparameters and the training data for QwQ-32B may lead to better performance, we leave the exploration to future work due to the heavy computation cost.

其次，对于 QwQ-32B 模型，我们观察到一个明显的临界阈值，在图 2中以竖直虚线标出。 超过这一阈值后，进一步缩短推理长度会导致性能急剧且显著地下降。 我们还发现，对 QwQ-32B 施加 4k 长度上限，结果甚至差于 3k 长度上限。 这一下降部分解释了，为什么迭代式剪枝在 QwQ-32B 上的表现，不如在 1.5B 等较小模型上。 尽管优化 QwQ-32B 的超参数和训练数据可能带来更好的性能，但由于计算成本较高，我们将这一探索留待未来工作。

对采用 ThinkPrune 的不同长思维链大语言模型应用预算强制后，其推理时思考长度与性能之间的权衡。

图3：对采用 ThinkPrune 的不同长思维链大语言模型应用预算强制后，其推理时思考长度与性能之间的权衡。

Inference-time trade-off with budget-forcing. We further study how different long-CoT LLMs perform when applying the budget-forcing method. Detailed budget-forcing prompt can be found in Appendix A.1. Figure 3 shows the trade-off between performance and output length on the Math-500 dataset (left) and the AIME-24 dataset (right), comparing models before and after applying ThinkPrune with budget-forcing. We highlight two key observations: First, ThinkPrune significantly improves performance under a limited thinking token budget. For example, ThinkPrune-4k→3k→2k consistently outperforms the original model when using the same number of thinking tokens.

使用预算强制时的推理时权衡。 我们进一步研究了对不同长思维链大语言模型应用预算强制方法时的性能。 详细的预算强制提示见附录 A.1。 图 3展示了 Math-500 数据集（左）和 AIME-24 数据集（右）上的性能与输出长度权衡，比较在预算强制条件下，应用 ThinkPrune 前后的模型。 我们重点指出两项观察结果：首先，ThinkPrune 显著提高了模型在有限思考 token 预算下的性能。 例如，在使用相同数量思考 token 时，ThinkPrune-4k→3k→2k 始终优于原始模型。

On Math-500, it reaches similar accuracy while using only about 50% of the original thinking tokens for Qwen1.5B-Distill-R1. This suggests that ThinkPrune helps the model think more efficiently and make better use of a limited token budget. Second, ThinkPrune reduces more thinking tokens on easier Math-500 problems than on AIME-24. This suggests that redundancy in reasoning is related to problem difficulty and that length pruning can adaptively remove unnecessary thinking for questions at different difficulties.

在 Math-500 上，对于 Qwen1.5B-Distill-R1，它只需使用原来约 50% 的思考 token，就能达到相近的准确率。 这说明，ThinkPrune 有助于模型更高效地思考，更好地利用有限的 token 预算。 其次，与 AIME-24 相比，ThinkPrune 在较简单的 Math-500 问题上能够减少更多思考 token。 这表明，推理冗余与问题难度有关，长度剪枝能够自适应地去除不同难度问题中不必要的思考。

4.3. Reasoning Behavior Analaysis

In this section, we take a closer look at the reasoning behavior of long-COT LLMs after length pruning. We focus on answering the following two questions:

本节进一步考察长思维链大语言模型在长度剪枝后的推理行为。 我们重点回答以下两个问题：

1. How does the reasoning behavior of long-COT LLMs change, and what gets removed most after pruning?
2. How does the pruning affect the readability of model-generated reasoning?

1. 长思维链大语言模型的推理行为如何变化，剪枝后哪些内容被去除得最多？
2. 剪枝如何影响模型生成推理过程的可读性？

Reasoning-related keyword frequency change. Figure 4 illustrates the frequency of specific reasoning-related keywords per response in the MATH500 dataset using the DeepSeek-R1-Distill-Qwen-1.5B model, both before and after applying our pruning method. Specifically, we count the number of occurrences of each keywords within the responses of LLMs and then normalize the count by the number of tokens in the responses, which gives us the frequency of each word in 1000 tokens.

推理相关关键词的频率变化。 图 4展示了 DeepSeek-R1-Distill-Qwen-1.5B 模型在 MATH500 数据集上的回答中，应用我们的剪枝方法前后，特定推理相关关键词的出现频率。 具体而言，我们统计各关键词在大语言模型回答中的出现次数，再除以回答的 token 数进行归一化，从而得到每 1,000 个 token 中各词的频率。

长度剪枝前后，与推理相关的关键词频率变化。

图4：长度剪枝前后，与推理相关的关键词频率变化。

The figure is divided into three sections. The left section contains phrases that signal hesitations or self-corrections, which we find undergo a significant drop in frequency after pruning. This indicates that the model would hesitate less.

该图分为三个部分。 左侧包含表示犹豫或自我纠正的短语，我们发现，剪枝后这些短语的频率显著下降。 这说明模型会更少犹豫。

The middle section contains phrases that signal self-verification, which undergo a slight drop in frequency after pruning. This indicates that the model would sometimes skip the self-verification steps, which may sacrifice a little bit of accuracy, to save the token length. Finally, the right section contains phrases that signal core computation and reasoning, which have a slight increase in frequency despite the overall reduction in length. This indicates that the model would maintain the reasoning process, because that is the core process that contributes to the final solution.

中间部分包含表示自我验证的短语，其频率在剪枝后略有下降。 这说明模型有时会跳过自我验证步骤，以节省 token 长度，但也可能因此牺牲少量准确率。 最后，右侧包含表示核心计算与推理的短语，尽管整体长度减少了，这些短语的频率仍略有上升。 这说明模型会保留推理过程，因为它是促成最终解答的核心环节。

长度剪枝前（左）与剪枝后（右）的推理行为变化。

图5：长度剪枝前（左）与剪枝后（右）的推理行为变化。

Reasoning behavior change. We further analyze the change in reasoning behaviors that the model utilizes to solve a problem. Specifically, we first prompt GPT-4o to summarize 9 frequent problem-solving phases, such as ‘understanding the problem’ and ‘applying known theorems/properties’ (complete list in Appendix A.2). Then, for each model-generated solution, we prompt GPT-4o to split the output into chunks and label each chunk with one of these phases. We measure the number of reasoning steps in each phase by counting segments separated by double newlines (“\n \n”).

推理行为变化。 我们进一步分析了模型在解题时所采用的推理行为如何变化。 具体而言，首先提示 GPT-4o 总结 9 种常见的解题阶段，例如“理解问题”和“应用已知定理或性质”，完整列表见附录 A.2。 随后，对于模型生成的每个解答，提示 GPT-4o 将输出划分为若干片段，并为每个片段标注其中一个阶段。 通过统计由两个换行符（“\n \n”）分隔的片段数量，我们衡量各阶段的推理步骤数。

Figure 5 shows the distribution of these reasoning steps before and after pruning. As can be observed, the model would spend less time on relatively redundant steps, such as ‘finalize and present the answer’, ‘reassess solution’, and ‘explore alternative approaches’. Meanwhile, we observe an increase in the percentage of core problem-solving steps, including ‘applying known theorems/properties’ and ‘computing or simplifying expressions’. These observations further confirm that the model would focus more on the core problem-solving steps and save on peripheral steps.

图 5展示了剪枝前后这些推理步骤的分布。 可以看到，模型会在相对冗余的步骤上花费更少时间，例如“整理并给出答案”“重新评估解答”和“探索其他方法”。 与此同时，我们观察到，核心解题步骤的占比有所上升，包括“应用已知定理或性质”和“计算或简化表达式”。 这些观察进一步证实，模型会更加关注核心解题步骤，并减少外围步骤的开销。

表2：不同模型的推理轨迹困惑度。

	PPL	Avg. Acc
DeepSeek-R1-1.5B-Distill
Original Model	1.91	82.9
ThinkPrune	1.90	83.2
DeepScaleR-1.5B-Preview
Original Model	1.95	88.5
ThinkPrune	2.02	86.9
QwQ-32B
Original Model	2.37	95.1
ThinkPrune	2.24	93.8

不同模型的推理轨迹困惑度。

Readability evaluation. One common concern in RLVR is that the heavy RL training may reduce the readability of the model's reasoning, resulting in mixed language or non-readable reasoning trace as shown in R1-Zero model. To examine whether ThinkPrune would also introduce similar issues, we measure the perplexity of generated reasoning traces on the Math-500 dataset for original model and ThinkPrune-4k→ 3k→ 2k using Qwen2.5-Math-7B. As shown in Table 2, our pruning method does not significantly affect reasoning readability — the perplexity remains nearly identical to the original model. Figure 6 in Appendix A.3 shows an example reasoning trace from the Distill-R1-1.5B LLM before and after pruning. As shown in the example, the original LLM repeatedly checks its previous reasoning multiple times, leading to heavy, redundant reasoning steps. On the contrary, the reasoning trace remains fully readable and focused on solving the problem after pruning, with more efficient problem-solving steps and only one self-verification step.

可读性评估。 RLVR 中一个常见的担忧是，大量强化学习训练可能降低模型推理过程的可读性，导致 R1-Zero 模型中所呈现的语言混杂或难以阅读的推理轨迹。 为考察 ThinkPrune 是否也会引入类似问题，我们使用 Qwen2.5-Math-7B，测量原始模型与 ThinkPrune-4k→3k→2k 在 Math-500 数据集上生成的推理轨迹的困惑度。 如表 2所示，我们的剪枝方法不会显著影响推理的可读性，其困惑度与原始模型几乎相同。 附录 A.3中的图 6给出了 Distill-R1-1.5B 大语言模型在剪枝前后的一个推理轨迹示例。 如示例所示，原始大语言模型会多次反复检查此前的推理，产生大量冗余推理步骤。 相比之下，剪枝后的推理轨迹仍然完全可读，并专注于解决问题，解题步骤更加高效，而且只进行一次自我验证。

Distill-R1-1.5B 在一道 Math-500 问题上的推理轨迹示例，比较长度剪枝前后的结果。

图6（附录）：Distill-R1-1.5B 在一道 Math-500 问题上的推理轨迹示例，比较长度剪枝前后的结果。

5. Conclusion

In this paper, we proposed ThinkPrune to reduce the reasoning length of long CoT LLMs. ThinkPrune introduces a length constraint during RL training, which discards unfinished thoughts and answers when sampling responses. To maintain model performance, we apply an iterative pruning strategy that gradually tightens the length limit over multiple rounds training. Experiments show that ThinkPrune reduces the reasoning length and achieves a strong performance-length trade-off. Further analysis shows that ThinkPrune effectively removes redundant steps while preserving key reasoning processes.

本文提出 ThinkPrune，用于减少长思维链大语言模型的推理长度。 ThinkPrune 在强化学习训练中引入长度约束，在采样回答时丢弃未完成的思考和答案。 为保持模型性能，我们采用迭代式剪枝策略，在多轮训练中逐步收紧长度上限。 实验表明，ThinkPrune 能够缩短推理长度，并实现良好的性能与长度权衡。 进一步分析表明，ThinkPrune 能够有效去除冗余步骤，同时保留关键推理过程。