---

Thinkless: LLM Learns When to Think

LLMReasoningRLNeurIPS 2025新加坡国立大学

Fang G, Ma X, Wang X. Thinkless: LLM learns when to think[C/OL]//Advances in Neural Information Processing Systems: Vol. 38. Curran Associates, Inc., 2025: 151268-151295[2026-11-04]. DOI:10.52202/085713-5060.

https://arxiv.org/abs/2505.13379v2

https://github.com/VainF/Thinkless

---

Thinkless：让大语言模型学会何时思考

Abstract

Reasoning Language Models, capable of extended chain-of-thought reasoning, have demonstrated remarkable performance on tasks requiring complex logical inference. However, applying elaborate reasoning for all queries often results in substantial computational inefficiencies, particularly when many problems admit straightforward solutions. This motivates an open question: Can LLMs learn when to think? To answer this, we propose Thinkless, a learnable framework that empowers an LLM to adaptively select between short-form and long-form reasoning, based on both task complexity and the model's ability. Thinkless is trained under a reinforcement learning paradigm and employs two control tokens, <short> for concise responses and <think> for detailed reasoning.

能够进行长思维链推理的推理语言模型，在需要复杂逻辑推断的任务上表现出色。 然而，对所有问题都进行详尽推理，往往会造成严重的计算低效，尤其是在许多问题本可以直接求解的情况下。 这引出了一个尚待回答的问题：大语言模型能否学会何时思考？ 为回答这一问题，我们提出 Thinkless，一个可学习的框架，使大语言模型能够根据任务复杂度和自身能力，自适应地选择短链或长链推理。 Thinkless 采用强化学习范式进行训练，并使用两个控制 token：<short> 用于简洁回答，<think> 用于详细推理。

At the core of our method is a Decoupled Group Relative Policy Optimization (DeGRPO) algorithm, which decomposes the learning objective of hybrid reasoning into two components: (1) a control token loss that governs the selection of the reasoning mode, and (2) a response loss that improves the accuracy of the generated answers. This decoupled formulation enables fine-grained control over the contributions of each objective, stabilizing training and effectively preventing collapse observed in vanilla GRPO. Empirically, on several benchmarks such as Minerva Algebra, MATH-500, and GSM8K, Thinkless is able to reduce the usage of long-chain thinking by 50% - 90%, significantly improving the efficiency of Reasoning Language Models. The code is available at https://github.com/VainF/Thinkless

我们的方法以解耦组相对策略优化（Decoupled Group Relative Policy Optimization，DeGRPO）算法为核心，将混合推理的学习目标分解为两个部分：（1）控制推理模式选择的控制 token 损失；（2）提高生成答案准确率的回答损失。 这种解耦形式能够精细控制各个目标的贡献，稳定训练，并有效防止原始 GRPO 中出现的坍塌现象。 实验表明，在 Minerva Algebra、MATH-500 和 GSM8K 等多个基准上，Thinkless 能将长链思考的使用量减少 50%–90%，显著提高推理语言模型的效率。 代码地址为 https://github.com/VainF/Thinkless

1. Introduction

Reasoning Language Models have exhibited notable effectiveness in solving complex tasks that involve multi-hop reasoning and logic-intensive inference. Their capabilities span a range of domains, such as mathematical problem solving and agentic assistants. A primary factor underlying this success is their ability to perform chain-of-thought reasoning, wherein intermediate steps are explicitly generated before arriving at a final answer. While this approach is effective for solving challenging problems, applying it uniformly to all queries, regardless of their complexity or the model’s capability can be inefficient. In particular, for questions with straightforward solutions, invoking extended reasoning results in redundant token generation, increased memory footprint, and substantially higher computational cost.

推理语言模型在解决涉及多跳推理和密集逻辑推断的复杂任务时，表现出了显著的有效性。 其能力覆盖数学解题和智能体助手等多个领域。 这一成功的主要原因之一是它们能够进行思维链推理，即在得出最终答案之前，显式生成中间步骤。 虽然这种方式能够有效解决困难问题，但不考虑问题复杂度或模型能力，对所有问题一律采用这种方式，可能效率低下。 特别是对于可以直接求解的问题，调用长篇推理会生成冗余 token，增加内存占用，并显著提高计算成本。

In response to these inefficiencies, a growing body of research has sought to enhance the inference efficiency of reasoning models. A prominent direction in this space explores hybrid reasoning, wherein models dynamically switch between reasoning and non-reasoning modes. Despite promising results, a central challenge persists: determining when a model should engage in elaborate reasoning. Many existing approaches address this by incorporating manually designed heuristics, such as fixed computational budgets or prompt-level control signals like “reasoning on/off”. However, these strategies inherently rely on human prior knowledge and may yield suboptimal or inappropriate control decisions. This underscores a fundamental open question: Can an LLM learn to decide when to think, guided by the complexity of the task and its own capability?

针对这些低效问题，越来越多的研究开始尝试提高推理模型的推理效率。 其中一个重要方向是探索 混合推理，让模型在推理与非推理模式之间动态切换。 尽管已有结果令人鼓舞，一个核心挑战仍然存在：如何判断模型应在何时进行详尽推理。 许多现有方法通过引入人工设计的启发式规则来解决这一问题，例如固定计算预算，或“推理开启/关闭”之类的提示级控制信号。 然而，这些策略本质上依赖人类先验知识，可能产生次优甚至不恰当的控制决策。 这进一步凸显了一个根本性的开放问题：大语言模型能否根据任务复杂度和自身能力，学会 决定何时思考？

Motivated by this, we explore the fundamental form of hybrid reasoning, where the model is tasked with autonomously deciding whether to generate a short-form or long-form response based on the input query. This decision is guided by three core factors: (1) the complexity of the query, as simpler questions generally merit concise responses, while more intricate ones may necessitate extended reasoning; (2) the capability of the model, since more powerful models are better positioned to employ short reasoning without sacrificing accuracy, whereas less capable models may benefit from longer responses to preserve performance; and (3) the user's tolerance for the trade-off between efficiency and accuracy, which determines the acceptable level of performance degradation when opting for shorter reasoning. Naturally, reinforcement learning offers a framework to unify these factors, as it allows the model to learn from interactions that reflect both environmental feedback and user-defined preferences. Through iterative exploration and reward-driven updates, the model progressively acquires the ability to make autonomous, context-aware decisions about its reasoning strategy, balancing accuracy and efficiency in a dynamic and data-driven manner.

受此启发，我们探索混合推理的一种基本形式：让模型根据输入问题，自主决定生成简短回答还是长篇回答。 这一决策由三个核心因素决定：（1）问题的复杂度，因为简单问题通常适合简洁回答，而更复杂的问题可能需要较长的推理；（2）模型的能力，因为更强的模型更有可能在不牺牲准确率的情况下采用短链推理，而能力较弱的模型则可能需要较长的回答来保持性能；（3）用户对效率与准确率之间取舍的容忍程度，它决定了选择较短推理时，可接受多大程度的性能下降。 强化学习自然地提供了一个统一这些因素的框架，因为它允许模型从同时反映环境反馈与用户定义偏好的交互中学习。 通过反复探索和奖励驱动的更新，模型逐步获得自主且感知上下文的推理策略决策能力，以动态、数据驱动的方式平衡准确率与效率。

Thinkless 学习一个混合大语言模型，通过两个特殊 token——<think> 和 <short>——自适应地选择思考与非思考推理模式。方法的核心是解耦组相对策略优化，它将控制 token 上的模式选择与回答 token 上的准确率提升分开，并平衡这两个目标。

图1：Thinkless 学习一个混合大语言模型，通过两个特殊 token——<think> 和 <short>——自适应地选择思考与非思考推理模式。方法的核心是解耦组相对策略优化，它将控制 token 上的模式选择与回答 token 上的准确率提升分开，并平衡这两个目标。

Building on these insights, we propose Thinkless, a reinforcement learning framework designed to train a hybrid reasoning model capable of selecting between short-form and long-form responses. As illustrated in Figure 3, Thinkless employs two control tokens, <think> and <short>, which are generated as the first token in the model’s output to signal the intended inference style. The training comprises two stages: a supervised warm-up phase followed by a reinforcement learning phase.

基于这些认识，我们提出 Thinkless，一个用于训练混合推理模型的强化学习框架，使模型能够在简短回答与长篇回答之间进行选择。 如图 3 所示，Thinkless 使用 <think> 和 <short> 两个控制 token，并将它们作为模型输出的第一个 token 生成，以指示所采用的推理方式。 训练包含两个阶段：先进行有监督预热，再进行强化学习。

Distillation for Warm-up. In the warm-up phase, the model aligns its response style with the designated control tokens via a distillation process. Specifically, it learns to imitate the behavior of two expert models: a reasoning model and a standard instruction-following model, each conditioned on a specific control token (<think> or <short>). Additionally, the model is trained on paired long-form and short-form responses for each query, ensuring it can generate both styles with comparable likelihood. This initialization establishes a clear and robust mapping between control tokens and response formats, providing diverse outputs for subsequent reinforcement learning.

用于预热的蒸馏。 在预热阶段，模型通过蒸馏，使其回答风格与指定的控制 token 对齐。 具体而言，模型学习模仿两个专家模型的行为：一个推理模型，以及一个标准的指令遵循模型，分别以特定的控制 token（<think> 或 <short>）为条件。 此外，每个问题都配有成对的长篇和简短回答用于训练，确保模型能够以相近的概率生成这两种风格的回答。 这一初始化过程在控制 token 与回答形式之间建立了清晰而稳健的映射，为后续强化学习提供多样化的输出。

Reinforcement Learning with Decoupled GRPO. In the reinforcement learning phase, the model is optimized to select the appropriate inference mode based on performance feedback. A natural starting point for this task is the vanilla Group Relative Policy Optimization (GRPO) framework. However, when applied to hybrid reasoning, vanilla GRPO treats all tokens, including the control token and the response tokens uniformly. This introduces a critical imbalance: since the response part often spans hundreds to thousands of tokens and the length of long & short responses varies significantly, the single control token may receive weak and biased gradient signals, ultimately leading to mode collapse at the early stages of training. To this end, we propose a tailored method for hybrid reasoning, termed Decoupled Group Relative Policy Optimization (DeGRPO).

使用解耦 GRPO 的强化学习。 在强化学习阶段，模型根据性能反馈进行优化，以选择合适的推理模式。 对于这项任务，一个自然的起点是原始的组相对策略优化（GRPO）框架。 然而，将原始 GRPO 应用于混合推理时，它会统一处理所有 token，包括控制 token 和回答 token。 这带来了一种关键失衡：回答部分往往包含数百到数千个 token，而且长、短回答的长度差异很大，因而单个控制 token 可能接收到微弱且有偏的梯度信号，最终在训练早期造成模式坍塌。 为此，我们提出一种专为混合推理设计的方法，称为解耦组相对策略优化（DeGRPO）。

As illustrated in Figure 1, DeGRPO explicitly separates the hybrid reasoning objective into two components: (1) Mode Selection, which governs how quickly the policy adapts based on the model’s current accuracy; and (2) Accuracy Improvement, which refines the response content to improve answer correctness under the selected reasoning mode. These two components are inherently interdependent, and effective training requires carefully balancing the learning signals for the control token and the response tokens. For instance, if the mode selection policy adapts too aggressively in favor of long-form reasoning, the model may ignore short-form responses, resulting in insufficient exploration of the potential of short responses. To this end, DeGRPO assigns distinct weights to the control token and response tokens, promoting a more stable and balanced training dynamic. This design not only mitigates mode collapse but also enables the model to learn both accurate outputs and context-sensitive reasoning strategies.

如图 1 所示，DeGRPO 将混合推理目标显式划分为两个部分：（1）模式选择，它决定策略根据模型当前准确率进行调整的速度；（2）准确率提升，它通过改进回答内容，提高所选推理模式下答案的正确性。 这两个部分本质上相互依赖，要实现有效训练，就必须谨慎平衡控制 token 与回答 token 的学习信号。 例如，如果模式选择策略过于激进地转向长篇推理，模型就可能忽略简短回答，从而无法充分探索简短回答的潜力。 为此，DeGRPO 为控制 token 和回答 token 分配不同的权重，促使训练过程更加稳定、均衡。 这种设计不仅缓解了模式坍塌，还使模型能够同时学习准确的输出和适应上下文的推理策略。

After reinforcement learning, the model learns to accurately identify simple queries and respond using the more efficient non-thinking mode. For instance, on benchmarks such as MATH-500, Minerva Algebra and GSM8K dataset, Thinkless reduces the usage of long-form reasoning by 50% - 90%. And on much more challenging tasks like AIME, the model naturally adopts a higher proportion of long-form reasoning.

经过强化学习后，模型学会准确识别简单问题，并使用效率更高的非思考模式作答。 例如，在 MATH-500、Minerva Algebra 和 GSM8K 等基准数据集上，Thinkless 将长篇推理的使用量减少了 50%–90%。 而在 AIME 这类更具挑战性的任务上，模型则会自然地提高长篇推理的使用比例。

In conclusion, this work demonstrates that a Reasoning Language Model can learn when to engage in reasoning before generating a response, guided by the proposed Decoupled GRPO method. This adaptive decision-making substantially reduces inference cost while preserving task performance.

总而言之，本研究表明，在所提出的解耦 GRPO 方法引导下，推理语言模型能够在生成回答之前，学会判断何时应进行推理。 这种自适应决策在保持任务性能的同时，大幅降低了推理成本。

2. Related Works

Efficient Reasoning Models. Reasoning models generate intermediate steps in a chain-of-thought process before producing a final answer. This paradigm offers significant advantages for tasks involving complex computations, logical deduction, and multi-step reasoning. However, excessively long reasoning chains can lead to substantial computational overhead. To mitigate this, recent research has explored strategies to enhance the efficiency of reasoning models without sacrificing accuracy or generalization.

高效推理模型。 推理模型会在给出最终答案前，通过思维链过程生成中间步骤。 对于涉及复杂计算、逻辑演绎和多步推理的任务，这一范式具有显著优势。 然而，过长的推理链会带来大量计算开销。 为缓解这一问题，近期研究探索了在不牺牲准确率或泛化能力的前提下，提高推理模型效率的策略。

Several techniques, such as reinforcement learning with length penalties, supervised fine-tuning using variable-length chain-of-thought data, and prompt-based methods have been proposed to encourage concise yet effective reasoning paths. Additionally, latent reasoning techniques aim to encode reasoning steps into compact internal representations, thereby reducing token-level computation while maintaining performance. Parallel efforts in knowledge distillation have facilitated the transfer of reasoning capabilities from large to smaller models, while efficient decoding strategies such as predictive decoding and self-consistency optimization have also yielded notable improvements in inference speed and resource utilization.

已有研究提出多种技术，以鼓励简洁而有效的推理路径，例如带长度惩罚的强化学习、使用可变长度思维链数据的有监督微调，以及基于提示的方法。 此外，潜在推理技术旨在将推理步骤编码为紧凑的内部表示，从而在保持性能的同时减少 token 层面的计算。 与此同时，知识蒸馏研究促进了推理能力从大模型向小模型的迁移，而预测式解码、自一致性优化等高效解码策略，也显著改善了推理速度与资源利用率。

Hybrid Reasoning. While prior work has largely focused on compressing reasoning paths to reduce token generation, an alternative path to efficiency is hybrid reasoning, which dynamically adapts the appropriate inference behaviour based on task complexity. This approach allows models to flexibly alternate between short-form responses and long-chain reasoning as needed. Hybrid reasoning can be realized either through collaborative systems involving multiple models or within a single unified model. In multi-model frameworks, routing mechanisms or speculative decoding techniques are commonly employed. For example, a lightweight model may generate a preliminary answer that a larger model verifies or refines.

混合推理。 以往工作主要关注通过压缩推理路径来减少 token 生成，而另一条提高效率的路径是混合推理，即根据任务复杂度，动态调整所采用的推理行为。 这种方法允许模型根据需要，在简短回答与长链推理之间灵活切换。 混合推理既可以通过多个模型协同的系统实现，也可以在单个统一模型中实现。 在多模型框架中，通常会采用路由机制或推测解码技术。 例如，可以由轻量模型生成初步答案，再由更大的模型进行验证或改进。

In contrast, unified models are trained to support both reasoning modes and can switch between them via prompt-based control. Some models adopt fixed prompt formats such as “reasoning on/off” to modulate reasoning depth. However, most existing approaches depend on manually crafted heuristics to balance efficiency and performance. In this work, we explore a learning-based alternative, enabling an LLM to automatically determine its inference behavior based on inputs, without relying on manual control.

相比之下，统一模型经过训练后能够支持两种推理模式，并通过基于提示的控制在两者之间切换。 一些模型采用“推理开启/关闭”等固定提示格式来调节推理深度。 然而，大多数现有方法依赖人工设计的启发式规则来平衡效率与性能。 在本研究中，我们探索一种基于学习的替代方案，使大语言模型能够根据输入自动决定推理行为，而无需依赖人工控制。

3. Method

The proposed Thinkless is implemented in two stages: (1) Distillation for Warm-up, where we fine-tune a pre-trained reasoning model to unify two reasoning styles, and (2) Reinforcement Learning with DeGRPO, where the model is trained under a decomposed objective to select the appropriate reasoning mode while improving the response quality.

所提出的 Thinkless 分两个阶段实现：（1）用于预热的蒸馏，通过微调预训练推理模型来统一两种推理风格；（2）使用 DeGRPO 的强化学习，在分解后的目标下训练模型，使其在提高回答质量的同时，选择合适的推理模式。

3.1. Distillation for Warm-up

The first step in our framework is to construct a model ${\pi }_{\theta }$ capable of generating both short- and long-form responses. We leverage two pre-trained experts for distillation: a reasoning model ${\pi }_{\text{think}}$, trained to produce detailed chains of thought via step-by-step reasoning; and an instruction-following model ${\pi }_{\text{short}}$, optimized for generating concise answers aligned with user intent.

我们框架的第一步是构建能够同时生成简短和长篇回答的模型 ${\pi }_{\theta }$。 我们使用两个预训练专家模型进行蒸馏：推理模型 ${\pi }_{\text{think}}$ 经过训练，能够通过逐步推理生成详细的思维链；指令遵循模型 ${\pi }_{\text{short}}$ 则针对生成符合用户意图的简洁答案进行了优化。

ThinkLess 训练一个根据任务复杂度和模型能力，自适应选择推理模式的混合模型。首先进行蒸馏，使模型能够遵循控制 token（<think> 或 <short>）来进行受控推理。随后使用解耦 GRPO 进行强化学习，将训练分为两个目标：优化控制 token，以实现有效的模式选择；改进回答，以提高答案准确率。

图2：ThinkLess 训练一个根据任务复杂度和模型能力，自适应选择推理模式的混合模型。首先进行蒸馏，使模型能够遵循控制 token（<think> 或 <short>）来进行受控推理。随后使用解耦 GRPO 进行强化学习，将训练分为两个目标：优化控制 token，以实现有效的模式选择；改进回答，以提高答案准确率。

Given a prompt corpus $\mathcal{X}=\{x_i{\}}_{i=1}^N$, we use these models to generate a synthetic paired dataset:

给定提示语料 $\mathcal{X}=\{x_i{\}}_{i=1}^N$，我们使用这些模型生成一个合成的配对数据集：

$$\begin{array}{rl}{\mathcal{D}}_{\text{distill}}=& \{(x_i,\mathtt{\text{<think>}}{a}_i^{\text{think}},\\ & \mathtt{\text{<short>}}{a}_i^{\text{short}}){\}}_{i=1}^N,\end{array}$$

where $a_i^{\text{think}}={\pi }_{\text{long}}(x_i)$ and $a_i^{\text{short}}={\pi }_{\text{short}}(x_i)$. Each response is prefixed with a control token $c\in \mathcal{C}=\{\mathtt{\text{<short>}},\mathtt{\text{<think>}}\}$ that conditions the model on the intended reasoning style. We then fine-tune the target reasoning model ${\pi }_{\theta }$ on this dataset via supervised fine-tuning (SFT). The objective is to learn a multi-style response distribution conditioned on the control token.

其中，$a_i^{\text{think}}={\pi }_{\text{long}}(x_i)$，$a_i^{\text{short}}={\pi }_{\text{short}}(x_i)$。 每个回答前都加上控制 token $c\in \mathcal{C}=\{\mathtt{\text{<short>}},\mathtt{\text{<think>}}\}$，以此为条件，让模型采用预期的推理风格。 随后，我们在这一数据集上对目标推理模型 ${\pi }_{\theta }$ 进行有监督微调（SFT）。 目标是学习一个以控制 token 为条件的多风格回答分布。

This distillation phase ensures that the model is capable of generating both types of responses with high fidelity. Moreover, the paired construction of ${\mathcal{D}}_{\text{distill}}$ ensures that, the model's response distribution will be balanced. This helps the follow-up RL process to explore different solutions.

这一蒸馏阶段确保模型能够高质量地生成这两类回答。 此外，${\mathcal{D}}_{\text{distill}}$ 的配对构造确保了模型回答分布的均衡。 这有助于后续强化学习过程探索不同的解答。

3.2. Learning When to Think via Decoupled GRPO

After the distillation phase, the model can produce both long‑ and short‑form answers. what it still lacks is a mechanism for deciding which reasoning mode suits a particular input $x$. To supply this capability we frame mode selection as a reinforcement‑learning problem and optimize a policy ${\pi }_{\theta }(c,a\mid x)={\pi }_{\theta }(c\mid x){\pi }_{\theta }(a\mid x,c)$, where the first token $c\in \mathcal{C}=\{\mathtt{\text{<short>}},\mathtt{\text{<think>}}\}$ serves as a control token that determines the reasoning mode, and the subsequent tokens $(a_{i,1},\dots ,a_{i,T_i})$ constitute the generated response. For notational convenience, we denote the entire sequence of the $i$-th sample of length $T_i+1$ as $a_i=(a_{i,0},\dots ,a_{i,T_i})$, where $a_{i,0}\in \mathcal{C}$ is the control token.

经过蒸馏阶段后，模型能够同时生成长篇和简短答案。 但它仍然缺少一种机制，来 决定 哪种推理模式适合特定的输入 $x$。 为赋予模型这一能力，我们将模式选择表述为强化学习问题，并优化策略 ${\pi }_{\theta }(c,a\mid x)={\pi }_{\theta }(c\mid x){\pi }_{\theta }(a\mid x,c)$，其中第一个 token $c\in \mathcal{C}=\{\mathtt{\text{<short>}},\mathtt{\text{<think>}}\}$ 作为控制 token 决定推理模式，而后续 token $(a_{i,1},\dots ,a_{i,T_i})$ 构成生成的回答。 为方便记号表示，我们将第 $i$ 个样本中长度为 $T_i+1$ 的完整序列记作 $a_i=(a_{i,0},\dots ,a_{i,T_i})$，其中 $a_{i,0}\in \mathcal{C}$ 是控制 token。

Reward Design. Let $y^{\ast }$ denote the ground-truth answer corresponding to the input $x$. We consider a minimally designed reward function $r(a,y^{\ast },c)$, which assigns different reward values as follows:

奖励设计。 令 $y^{\ast }$ 表示输入 $x$ 对应的真实答案。 我们考虑一个尽量简洁的奖励函数 $r(a,y^{\ast },c)$，按如下方式赋予不同的奖励值：

$$\begin{array}{rl}& r(a,y^{\ast },c)=\\ & \{\begin{array}{ll}1.0,& \begin{array}{rl}& \text{if }c=\text{<short>}\text{ and}\\ & \text{Extract-Answer}(a)=y^{\ast },\end{array}\\ 1.0-\gamma ,& \begin{array}{rl}& \text{if }c=\text{<think>}\text{ and}\\ & \text{Extract-Answer}(a)=y^{\ast },\end{array}\\ -1.0,& \text{if }\text{Extract-Answer}(a)\ne y^{\ast },\end{array}\end{array}$$

where the $1>\gamma >0$ introduces preference to the short correct answer over long responses.

其中，$1>\gamma >0$ 使正确的简短答案比长篇回答更受偏好。

Decoupled Policy Optimization. Based on the simple reward function, we adopt a GRPO-based framework for training. Let $\{a_i{\}}_{i=1}^G$ denote a mini‑batch of trajectories sampled from the current policy ${\pi }_{{\theta }_{\mathrm{old}}}$. The objective is defined as:

解耦策略优化。 基于这一简单的奖励函数，我们采用基于 GRPO 的框架进行训练。 令 $\{a_i{\}}_{i=1}^G$ 表示从当前策略 ${\pi }_{{\theta }_{\mathrm{old}}}$ 中采样得到的一小批轨迹。 目标定义为：

$$\begin{array}{}\text{(1)}& \begin{array}{rl}& {\mathcal{J}}_{\mathrm{GRPO}}(\theta )=\\ & {\mathbb{E}}_{x,a_i}[\frac{1}{G}\sum _{i=1}^G(\frac{1}{T_i+1}\sum _{t=0}^{T_i}{\mathcal{L}}_{i,t}(\theta )\\ & -\beta {\mathbb{D}}_{\mathrm{KL}}[{\pi }_{\theta }(\cdot \mid x)\parallel {\pi }_{\mathrm{ref}}(\cdot \mid x)])],\end{array}\end{array}$$

where ${\mathcal{L}}_{i,t}(\theta )$ denotes the token-level surrogate loss formally given by:

其中，${\mathcal{L}}_{i,t}(\theta )$ 表示 token 级的代理损失，其形式为：

$$\begin{array}{}\text{(2)}& \begin{array}{rl}& {\mathcal{L}}_{i,t}(\theta )=\\ & min(\frac{{\pi }_{\theta }(a_{i,t}\mid x,a_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(a_{i,t}\mid x,a_{i,<t})}{\hat{A}}_{i,t},\\ & \text{clip}(\frac{{\pi }_{\theta }(a_{i,t}\mid x,a_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(a_{i,t}\mid x,a_{i,<t})},\\ & 1-ϵ,1+ϵ){\hat{A}}_{i,t})\end{array}\end{array}$$

In this work, we compute the relative advantage using ${\hat{A}}_{i,t}=r-\mathrm{mean}(\mathbf{r})$, following Liu et al. This choice is motivated by the observation that our training data contains questions of varying difficulty, which can introduce bias when using standard deviation normalization.

在本研究中，我们遵循 Liu 等人的做法，使用 ${\hat{A}}_{i,t}=r-\mathrm{mean}(\mathbf{r})$ 计算相对优势。 之所以采用这一选择，是因为我们观察到，训练数据包含难度各异的问题，使用标准差归一化可能引入偏差。

When applied to training, the objective in Equation 1 serves two purposes: (1) to learn an appropriate control token for mode selection, and (2) to improve the accuracy of the response tokens:

用于训练时，公式 1 中的目标具有两个作用：（1）学习合适的控制 token，以进行模式选择；（2）提高回答 token 的准确性：

$$\begin{array}{}\text{(3)}& \begin{array}{rl}& \frac{1}{T_i+1}\sum _{t=0}^{T_i}{\mathcal{L}}_{i,t}(\theta )\\ & =\underset{\text{Control Token}}{\underbrace{\frac{1}{T_i+1}{\mathcal{L}}_{i,0}(\theta )}}\\ & +\underset{\text{Response Tokens}}{\underbrace{\frac{1}{T_i+1}\sum _{t=1}^{T_i}{\mathcal{L}}_{i,t}(\theta )}}.\end{array}\end{array}$$

For mode selection, the response style is conditioned on the first token $a_{i,0}$, which is trained during the preceding distillation stage. As a result, adjusting the probability of this single control token is sufficient to switch between reasoning modes. Therefore, this token controls the learning of inference mode. For response accuracy, the optimization seeks to improve the generation of the remaining tokens $a_{i,1:T_i}$.

对于模式选择，回答风格以第一个 token $a_{i,0}$ 为条件，而这一 token 已在前面的蒸馏阶段得到训练。 因此，只需调整这一个控制 token 的概率，就足以在推理模式之间切换。 由此，这一 token 控制着推理模式的学习。 对于回答准确率，优化的目的是改善其余 token $a_{i,1:T_i}$ 的生成。

However, the above Equation 3 introduces two types of imbalance during optimization: (1) Mode-Accuracy imbalance - each trajectory contains only one control token but $T_i$ response tokens, disproportionately reducing the influence of the mode selection compared to the optimization of response accuracy. (2) Think-Short imbalance - longer sequences ($T_i^{\text{think}}\gg T_i^{\text{short}}$) further suppress the gradient contribution of the control token due to the normalization factor $1/(T_i+1)$, causing the <think> token to be under-optimized compared to the <short>. As we will show in the experiment, such imbalance may lead to severe mode-collapse at the beginning of training. To mitigate these imbalances, we propose a decoupled variant of GRPO, denoted as ${\mathcal{J}}_{\mathrm{DeGRPO}}$, which separately normalizes the contributions of the control and response tokens:

然而，上述公式 3 会在优化过程中引入两类失衡：（1）模式与准确率失衡——每条轨迹只有一个控制 token，却有 $T_i$ 个回答 token，使模式选择的影响相对于回答准确率的优化被不成比例地削弱。 （2）思考与简短回答失衡——较长的序列（$T_i^{\text{think}}\gg T_i^{\text{short}}$）会因归一化因子 $1/(T_i+1)$ 而进一步压低控制 token 的梯度贡献，导致 <think> token 相对于 <short> 得不到充分优化。 我们将在实验中展示，这种失衡可能在训练初期引发严重的模式坍塌。 为缓解这些失衡，我们提出 GRPO 的一种解耦变体，记为 ${\mathcal{J}}_{\mathrm{DeGRPO}}$，分别归一化控制 token 和回答 token 的贡献：

$$\begin{array}{}\text{(4)}& \begin{array}{rl}& {\mathcal{J}}_{\mathrm{DeGRPO}}(\theta )=\\ & {\mathbb{E}}_{x,a_i}[\frac{1}{G}\sum _{i=1}^G(\underset{\text{Control Token}}{\underbrace{\alpha {\mathcal{L}}_{i,0}(\theta )}}\\ & +\underset{\text{Response Tokens}}{\underbrace{\frac{1}{T_i}\sum _{t=1}^{T_i}{\mathcal{L}}_{i,t}(\theta )}}\\ & -\beta {\mathbb{D}}_{\mathrm{KL}}[{\pi }_{\theta }(\cdot \mid x)\parallel {\pi }_{\mathrm{ref}}(\cdot \mid x)])],\end{array}\end{array}$$

In DeGRPO, the mode selection ${\mathcal{L}}_{i,0}(\theta )$ and response accuracy improvement $\sum _{t=1}^{T_i}{\mathcal{L}}_{i,t}(\theta )$ are independently normalized. A length-independent weighting coefficient $\alpha$ is introduced to balance the optimization between mode selection and response generation. This formulation ensures that the control token receives a consistent gradient scale across both short and long sequences, thereby addressing the mode-mode and think-short imbalance, enabling more stable optimization of reasoning-mode selection. As will be shown in the experiments, an appropriately large $\alpha$ can make the mode update more efficient. In our experiment, we set $\alpha =1/1000$ for stable training.

在 DeGRPO 中，模式选择 ${\mathcal{L}}_{i,0}(\theta )$ 与回答准确率提升 $\sum _{t=1}^{T_i}{\mathcal{L}}_{i,t}(\theta )$ 分别进行归一化。 我们引入一个与长度无关的加权系数 $\alpha$，以平衡模式选择和回答生成之间的优化。 这一形式确保控制 token 在短序列和长序列中都接收到一致的梯度尺度，从而解决前述两类失衡，使推理模式选择的优化更加稳定。 实验将表明，适当增大 $\alpha$ 可以提高模式更新的效率。 在我们的实验中，为稳定训练，设置 $\alpha =1/1000$。

Method Summary. In summary, our method retains the overall structure of the standard GRPO framework. For each query, a mini-batch of samples is drawn from the current policy to estimate the token-level advantages. To address the imbalance between mode selection and response generation, we independently normalize the advantages associated with the control token and response tokens. This separation allows for explicit balancing of their contributions during optimization, leading to more stable and effective training.

方法小结。 总体而言，我们的方法保留了标准 GRPO 框架的整体结构。 对于每个问题，从当前策略中抽取一小批样本，以估计 token 级优势。 为解决模式选择与回答生成之间的失衡，我们对控制 token 和回答 token 对应的优势分别进行归一化。 这种分离使我们能够在优化期间显式平衡两者的贡献，从而实现更稳定、更有效的训练。

4. Experiments

4.1. Experimental Setups

LLMs and Datasets. We employ DeepSeek-R1-Distill-Qwen-1.5B as the base model to train a hybrid reasoning policy. To construct long-short paired responses for the distillation phase, we utilize long-form data from open-source datasets generated by the DeepSeek-R1-671B model, which is well-suited for multi-step reasoning. The corresponding short-form answers are derived using Qwen2.5-Math-1.5B-Instruct, a compact instruction-tuned model optimized for concise mathematical responses. The hybrid model is direcytly fine-tuned on this paired dataset via supervised fine-tuning, enabling it to accommodate both long and short reasoning styles.

大语言模型与数据集。 我们采用 DeepSeek-R1-Distill-Qwen-1.5B 作为基础模型，训练混合推理策略。 为构建蒸馏阶段的长短配对回答，我们使用开源数据集中的长篇数据，这些数据由擅长多步推理的 DeepSeek-R1-671B 模型生成。 相应的简短答案由 Qwen2.5-Math-1.5B-Instruct 生成，这是一个针对简洁数学回答进行优化的小型指令微调模型。 通过有监督微调，直接在这一配对数据集上微调混合模型，使其能够同时支持长链与短链推理风格。

The model is then further optimized using the Decoupled Group Relative Policy Optimization (GRPO) algorithm. For the reinforcement learning stage, we primarily use the DeepScaleR dataset, which comprises approximately 40K labeled examples. For evaluation, we mainly focus on math datasets, including AIME, Minerva Algebra, MATH-500 and GSM-8K.

随后，使用解耦组相对策略优化（GRPO）算法进一步优化模型。 在强化学习阶段，我们主要使用包含约 4 万个带标签示例的 DeepScaleR 数据集。 评估主要围绕数学数据集展开，包括 AIME、Minerva Algebra、MATH-500 和 GSM-8K。

Training Details. All experiments were conducted on a single node with 4 H100 GPUs. For the warmup stage, we set the maximum context length to 16K and clip those overlong samples. We train the DeepSeek-R1-Distill-Qwen-1.5B for only 1 epoch. The SFT was conducted on the Megatron framework. For the reinforcement learning stage, we extend the context length to 24K.

训练细节。 所有实验均在配有 4 块 H100 GPU 的单个节点上进行。 在预热阶段，我们将最大上下文长度设为 16K，并截断超长样本。 我们仅对 DeepSeek-R1-Distill-Qwen-1.5B 训练 1 个 epoch。 SFT 使用 Megatron 框架完成。 在强化学习阶段，我们将上下文长度扩展至 24K。

The model was trained only for 600 steps, using the AdamW optimizer with a learning rate of $1\times {10}^{-6}$, $\beta =(0.9,0.999)$, and a weight decay of 0.01. The batch size is set to 128, with 8 responses sampled for each query, leading to 1024 data points in total. The RL experiments were implemented using the VeRL framework. More details can be found in the Appendix.

模型仅训练了 600 步，使用 AdamW 优化器，学习率为 $1\times {10}^{-6}$，$\beta =(0.9,0.999)$，权重衰减为 0.01。 批大小设为 128，每个问题采样 8 个回答，因此总计得到 1024 个数据点。 强化学习实验使用 VeRL 框架实现。 更多细节见附录。

表1：混合推理的实验结果。对于混合算法，我们额外报告了评估时采用思考模式的问题比例。

Models	AIME 2024		Minerva Algebra		Math-500		GSM8K
	Pass@1	#Tokens (Think%)	Pass@1	#Tokens (Think%)	Pass@1	#Tokens (Think%)	Pass@1	#Tokens (Think%)
Base LLM
DeepSeek-R1-1.5B	0.2800	18063	0.9577	3029	0.8608	5675	0.8347	1919
Q-1.5B	0.0200	1300	0.7771	933	0.5168	855	0.7022	466
QMath-1.5B	0.1133	1128	0.9184	586	0.7604	721	0.8572	447
Short CoT
Merging-0.5	0.1333	8636	0.9292	834	0.7740	1524	0.8332	601
Merging-0.6	0.1733	10615	0.9321	1091	0.7900	3000	0.8381	747
Merging-0.7	0.1667	15854	0.9398	1834	0.8108	4347	0.8458	1201
CoT-Valve α=8	0.2000	10692	0.8079	1903	0.7060	3723	0.7726	773
CoT-Valve α=6	0.1933	17245	0.9468	2656	0.8024	5167	0.7970	1009
CoT-Valve α=4	0.2267	17722	0.9439	2965	0.8036	5820	0.8108	1396
Hybrid
Router Random	0.1467	8093 (56.00%)	0.9211	1736 (49.28%)	0.7608	3096 (47.92%)	0.8205	1086 (50.99%)
Router Q-7B	0.1667	9296 (46.67%)	0.9250	795 (5.64%)	0.7948	2748 (25.00%)	0.8587	563 (2.35%)
Thinkless	0.2733	7099 (100.00%)	0.9459	1144 (25.88%)	0.8184	2555 (51.56%)	0.8418	624 (13.31%)

混合推理的实验结果。对于混合算法，我们额外报告了评估时采用思考模式的问题比例。

4.2. Empirical Results on Hybrid Reasoning

Finding 1. The learned hybrid reasoning models effectively distinguish complex from simple queries, reducing the use of thinking by 50%–90%.

发现 1。 学到的混合推理模型能够有效区分复杂问题与简单问题，将思考模式的使用量减少 50%–90%。

Table 1 presents a comparison between our method and several existing models or techniques. The first part showcases our baseline model, DeepSeek-R1-Distill-Qwen-1.5B, alongside two instruction-following models designed to generate concise answers. It can be observed that the number of tokens generated by reasoning models is typically 5 to 20 times higher than that of standard models. This highlights the potential for significantly improving efficiency by appropriately selecting the reasoning mode. Notably, on challenging datasets such as AIME and MATH-500, reasoning models tend to outperform others by a large margin. However, on simpler datasets like GSM-8K, extended reasoning offers no clear advantage, and the Qwen2.5-Math-1.5B-Instruct model with only a 4K context length achieves better results.

表 1 比较了我们的方法与若干现有模型或技术。 第一部分展示了我们的基线模型 DeepSeek-R1-Distill-Qwen-1.5B，以及两个旨在生成简洁答案的指令遵循模型。 可以看到，推理模型生成的 token 数通常是标准模型的 5–20 倍。 这凸显出通过恰当选择推理模式来显著提高效率的潜力。 值得注意的是，在 AIME 和 MATH-500 等较难的数据集上，推理模型往往大幅优于其他模型。 然而，在 GSM-8K 等较简单的数据集上，长篇推理没有明显优势，上下文长度仅为 4K 的 Qwen2.5-Math-1.5B-Instruct 模型反而取得了更好的结果。

The second part of the table illustrates techniques for generating shorter chains of thought. One highly effective approach is model merging, where DeepSeek-R1-Distill-Qwen-1.5B is interpolated in parameter space with a base model, Qwen2.5-Math-1.5B, to obtain a more efficient model without additional training. Another method is the CoT-Valve technique, which applies supervised fine-tuning (SFT) using LoRA. It allows for controllable reasoning length by adjusting the $\alpha$ parameter in LoRA to modulate the magnitude of parameter updates.

表格的第二部分展示了生成较短思维链的技术。 一种非常有效的方法是模型合并，即在参数空间中对 DeepSeek-R1-Distill-Qwen-1.5B 与基础模型 Qwen2.5-Math-1.5B 进行插值，从而无需额外训练即可得到更高效的模型。 另一种方法是 CoT-Valve，它使用 LoRA 进行有监督微调（SFT）。 该方法通过调整 LoRA 中的 $\alpha$ 参数来调节参数更新幅度，从而控制推理长度。

Both methods provide mechanisms for adjusting reasoning length, such as the interpolation ratio and the LoRA $\alpha$. To show their performance, we sample outputs across a range of lengths. A notable challenge with these methods is that the optimal reasoning length varies significantly across datasets. Consequently, when a good heuristic, such as merging coefficients of 0.6, is determined on one dataset to reduce token usage, it may result in unexpected performance degradation on other benchmarks, such as AIME.

这两种方法都提供了调整推理长度的机制，例如插值比例和 LoRA 的 $\alpha$。 为展示其性能，我们在不同长度设置下采样输出。 这些方法面临的一个显著挑战是，最优推理长度在不同数据集之间差异很大。 因此，当在某个数据集上确定了较好的启发式设置，例如使用 0.6 的合并系数来减少 token 用量时，却可能在 AIME 等其他基准上造成意料之外的性能下降。

In the final part of our study, we examine hybrid reasoning strategies, focusing on a comparison with router-based approaches. These methods employ a separate large language model (LLM) to assess query difficulty and dispatch inputs to either a reasoning model or a standard model. While effective to some extent, router models are typically independent and lack comprehensive awareness of the target reasoning models, limiting their ability to make confidence-informed decisions. For instance, on the challenging AIME dataset, where even the reasoning model achieves only 28% accuracy, the router struggles to recognize the its difficulty.

在研究的最后一部分，我们考察混合推理策略，重点与基于路由器的方法进行比较。 这些方法采用独立的大语言模型来评估问题难度，并将输入分配给推理模型或标准模型。 虽然在一定程度上有效，但路由模型通常是独立的，对目标推理模型缺乏全面认识，因而难以根据置信度作出决策。 例如，在困难的 AIME 数据集上，即使推理模型的准确率也只有 28%，路由器却难以识别其难度。

In contrast, our method jointly considers both input complexity and model capability, dynamically refining the dispatch strategy through direct interaction with real examples. As a result, it achieves efficient and adaptive reasoning without manual tuning. On the Minerva Algebra dataset, our approach activates the reasoning mode for only 25% of the samples, reducing token usage to one-third of the original, while maintaining performance within a 1% margin. In addition, we found that the RL will also compress the length of long responses, since the algorithm will encourage short and correct answers, producing a gradient towards a more compact response.

相比之下，我们的方法同时考虑输入复杂度和模型能力，通过与真实示例的直接交互，动态改进分配策略。 因此，无需人工调参，即可实现高效且自适应的推理。 在 Minerva Algebra 数据集上，我们的方法仅对 25% 的样本启用推理模式，将 token 用量降至原来的三分之一，同时将性能差距控制在 1% 以内。 此外，我们发现强化学习还会压缩长篇回答的长度，因为该算法会鼓励简短且正确的答案，产生使回答更加紧凑的梯度。

4.3. Training Dynamics in RL

Finding 2. Policy may collapse due to imbalanced update of control tokens in Vanilla GRPO.

发现 2。 原始 GRPO 中控制 token 的更新失衡，可能导致策略坍塌。

Mode Collapse in RL. To further analyze how the model learns a reasonable policy, we visualize the training process of RL. Figure 3 (a) illustrates the Mode Collapse issue in standard GRPO, where the model develops an excessive preference for either long or short outputs during training. In conventional GRPO, the gradient on the control token is normalized by the total length of the response, which introduces an imbalance between long and short outputs. Specifically, long-chain samples, due to having more tokens, receive slower updates on the <think> token, while samples encouraging <short> dominate the updates. This imbalance causes the model to collapse rapidly, as shown in Figure 3 (a), the number of generated long-chain responses drops below 10 within just 120 update steps, making it difficult for the model to learn the correct policy. Furthermore, as shown in Figure 3 (c), the model fails to correctly differentiate between samples of varying difficulty, consistently opting for the short-chain reasoning mode.

强化学习中的模式坍塌。 为进一步分析模型如何学会合理的策略，我们将强化学习训练过程可视化。 图 3（a）展示了标准 GRPO 中的 模式坍塌 问题，即模型在训练期间对长输出或短输出中的一种产生过度偏好。 在常规 GRPO 中，控制 token 的梯度按照回答的总长度进行归一化，这会在长输出和短输出之间引入失衡。 具体而言，长链样本由于包含更多 token，其 <think> token 的更新更慢，而鼓励 <short> 的样本则主导了更新。 这种失衡使模型迅速坍塌：如图 3（a）所示，仅经过 120 个更新步骤，生成的长链回答数量就降至 10 以下，使模型难以学到正确的策略。 此外，如图 3（c）所示，模型无法正确区分难度各异的样本，而是始终选择短链推理模式。

原始 GRPO 与解耦 GRPO 的策略训练过程对比。

图3：原始 GRPO 与解耦 GRPO 的策略训练过程对比。

Finding 3. The U-shape learning curve: The proportion of short-chain samples first drops due to low initial accuracy, then rises after accuracy improvement and mode selection take effect.

发现 3。 U 形学习曲线：由于初始准确率较低，短链样本的比例先下降；随着准确率提升和模式选择发挥作用，这一比例随后回升。

The U-Shape Learning Curve. To mitigate the collapse issue, we propose a Decoupled GRPO algorithm. Figure 3 (b) illustrates the impact of this decoupling on the training process. We observe a characteristic U-shaped curve in the RL process: the proportion of long-chain outputs initially increases and then gradually decreases. After properly balancing the update weights between the control token and response tokens, the model shows a preference for long-chain reasoning in the early stages of training, primarily because long chains tend to yield higher accuracy. As training progresses, we observe an improvement in the accuracy of short-chain responses.

U 形学习曲线。 为缓解坍塌问题，我们提出了解耦 GRPO 算法。 图 3（b）展示了这种解耦对训练过程的影响。 我们在强化学习过程中观察到一种典型的 U 形曲线：长链输出的比例先上升，然后逐渐下降。 在恰当平衡控制 token 与回答 token 的更新权重后，模型在训练早期表现出对长链推理的偏好，主要因为长链通常能够取得更高的准确率。 随着训练推进，我们观察到短链回答的准确率不断提高。

This is driven by two factors: (1) reinforcement learning enhances the generation quality, and (2) the model learns to assign simpler queries to the short-chain reasoning mode. As a result, short-chain responses receive increasingly higher rewards, encouraging the model to further explore the feasibility of short reasoning. This behavior is manifested in the rising proportion of short-chain outputs over time, with many short responses achieving perfect accuracy in the later stages of training. Additionally, we observe a decline in the accuracy of long-chain responses during the latter half of training. This phenomenon is not due to a degradation of the model’s reasoning ability but rather a shift in task allocation: more difficult queries, which cannot be solved via short reasoning, are assigned to the long-chain mode, thereby lowering its average accuracy.

这一变化由两个因素驱动：（1）强化学习提高了生成质量；（2）模型学会将更简单的问题分配给短链推理模式。 因此，短链回答获得越来越高的奖励，促使模型进一步探索短链推理的可行性。 这种行为表现为短链输出的比例随时间上升，并且在训练后期，许多简短回答实现了完全正确。 此外，我们观察到，长链回答的准确率在训练的后半程有所下降。 这一现象并非模型推理能力退化所致，而是任务分配发生了变化：无法通过短链推理解决的较难问题被分配给长链模式，从而拉低了其平均准确率。

Finding 4. The weight $\alpha$ of the control token governs the learning speed of mode selection.

发现 4。 控制 token 的权重 $\alpha$ 决定了模式选择的学习速度。

较大的 token 损失系数 $\alpha$ 会加快推理行为的转变，使短模式下全部回答正确的样本迅速出现。

图4：较大的 token 损失系数 $\alpha$ 会加快推理行为的转变，使短模式下全部回答正确的样本迅速出现。

The Influence of Decoupling. Building upon the above analysis, we further visualize the effect of decoupling on model behavior. In Figure 4, we show the number of samples that are correctly answered with short responses across all sampled trajectories. We compare the results under two settings: a high control token update weight (0.5) and the weight used in our method (0.001). We observe that with a higher weight on the control token, the all-correct short samples emerge earlier in training.

解耦的影响。 基于上述分析，我们进一步将解耦对模型行为的影响可视化。 图 4 展示了在所有采样轨迹中，均能用简短回答正确作答的样本数量。 我们比较了两种设置的结果：较高的控制 token 更新权重（0.5），以及本方法采用的权重（0.001）。 我们观察到，控制 token 的权重较高时，短模式下全部回答正确的样本会在训练中更早出现。

This is because the control token is updated more aggressively, allowing the model to focus more on learning the mode selection. However, excessively fast policy updates can be problematic. For example, some samples may initially yield low accuracy under short responses, but reinforcement learning could eventually improve their short-mode performance. If the model’s strategy updates too quickly, it tends to assign such samples prematurely to the long-chain mode, degenerating the algorithm to a simple binary classifier based on initial accuracy rather than a collaborative learning of mode selection and accuracy improvement.

这是因为控制 token 更新得更加激进，使模型更加侧重于学习模式选择。 然而，过快的策略更新可能带来问题。 例如，某些样本最初采用简短回答时准确率较低，但强化学习最终可能改善它们在短模式下的表现。 如果模型策略更新过快，就容易过早地把这类样本分配给长链模式，使算法退化为一个基于初始准确率的简单二分类器，而不是协同学习模式选择与准确率提升。

4.4. Details of Warm-up Distillation

表2：预热阶段不同 SFT 数据集的有效性。由于这些模型尚未经过强化学习优化，因此手动插入控制 token <think> 和 <short>，以引出所需的回答模式。

Model	Mode & Teacher	AIME 2024		Minerva Algebra		Math-500		GSM8K
		Pass@1	#Tokens	Pass@1	#Tokens	Pass@1	#Tokens	Pass@1	#Tokens
Qwen2.5-1.5B-Instruct	Short (Base)	0.0200	1300	0.7771	933	0.5168	855	0.7022	466
Qwen2.5-Math-1.5B-Instruct	Short (Base)	0.1133	1128	0.9184	586	0.7604	721	0.8572	447
DeepSeek-R1-1.5B	Long (Base)	0.2800	18063	0.9577	3029	0.8608	5675	0.8347	1919
R1-1.5B + OpenR1-97K	◆ Long (R1-671B)	0.2933	18880	0.9456	3239	0.8336	6780	0.8382	3423
	◇ Short (QMath-1.5B)	0.0733	3023	0.9001	642	0.7212	1004	0.7985	433
R1-1.5B + OpenThoughts-114K	◆ Long (R1-671B)	0.2600	17964	0.9506	3122	0.8280	6430	0.8039	2427
	◇ Short (QMath-1.5B)	0.0800	3807	0.8947	702	0.7136	1004	0.7886	449
R1-1.5B + OpenThoughts-1M	◆ Long (R1-671B)	0.3067	18685	0.9530	3064	0.8360	6209	0.8253	2363
	◇ Short (QMath-1.5B)	0.0800	3977	0.9051	690	0.7276	989	0.8202	441

预热阶段不同 SFT 数据集的有效性。由于这些模型尚未经过强化学习优化，因此手动插入控制 token <think> 和 <short>，以引出所需的回答模式。

Finding 5. Reasoning LLMs can be a good short response learner.

发现 5。 推理大语言模型也能很好地学会简短回答。

In this work, knowledge distillation is deployed for warm-up, serving as a critical step to equip the model with the basic ability to generate both long chains and short responses. In this section, we provide additional implementation details. Specifically, we consider multiple datasets for distillation. We compare three datasets of increasing scale and domain coverage: (1) OpenR1, a mathematics-only dataset with rigorously verified solutions; (2) OpenThoughts-114K, a compact yet multi-domain dataset labeled by DeepSeek-R1-67B, covering mathematics, science, and programming; and (3) OpenThoughts-1M, a large-scale and diverse collection that subsumes the former two. Table 2 presents the training results on these datasets. Notably, we find that generating short responses using a long-chain reasoning model is relatively straightforward; even with the smallest dataset, OpenR1-97K, the target model successfully learns to produce short outputs.

在本研究中，我们使用知识蒸馏进行预热；这一关键步骤赋予模型同时生成长链与简短回答的基本能力。 本节提供更多实现细节。 具体而言，我们考虑了多个用于蒸馏的数据集。 我们比较了三个规模与领域覆盖范围逐步增大的数据集：（1）OpenR1，一个仅包含数学问题、且解答经过严格验证的数据集；（2）OpenThoughts-114K，一个由 DeepSeek-R1-67B 标注、规模较小但涵盖多个领域的数据集，包括数学、科学和编程；（3）OpenThoughts-1M，一个规模大、内容多样的数据集合，包含前述两个数据集。 表 2 给出了在这些数据集上的训练结果。 值得注意的是，我们发现，让长链推理模型生成简短回答相对容易：即使使用最小的 OpenR1-97K 数据集，目标模型也能成功学会生成简短输出。

However, this distillation process may incur some performance degradation. For instance, on the Math-500 benchmark, the accuracy of the distilled hybrid model is slightly lower than that of the original model, which may affect the downstream performance during the RL phase. For the distillation stage, larger and more comprehensive datasets can lead to improved performance. However, the marginal gains diminish as the dataset size increases. For example, expanding the dataset from 114K to 1M results in only a 1% improvement in long-chain accuracy on the Math-500 benchmark. This work provides a preliminary validation of the effectiveness of simple distillation, and we leave the construction of stronger initial hybrid models as an important direction for future research.

然而，这一蒸馏过程可能造成一定程度的性能下降。 例如，在 Math-500 基准上，蒸馏得到的混合模型准确率略低于原始模型，这可能影响其在强化学习阶段的后续性能。 在蒸馏阶段，更大且更全面的数据集能够改善性能。 但随着数据集规模增大，边际收益会逐渐减小。 例如，将数据集从 114K 扩展至 1M，在 Math-500 基准上仅带来了 1% 的长链准确率提升。 本研究初步验证了简单蒸馏的有效性，并将构建更强的初始混合模型留作未来研究的重要方向。

4.5. Case Study

Figure 5 presents a case study on the model’s predicted probability of selecting the <think> token across the MATH-500 dataset. The distribution reveals that the model makes smooth and hierarchical predictions for queries of different difficulty. In addition, we highlight representative examples corresponding to high, medium, and low confidence levels to illustrate the model’s decision behavior. It can be observed that samples assigned to the short reasoning mode are typically simple arithmetic problems that do not require deep or complex reasoning. In contrast, questions routed to the thinking mode tend to be more complex, involving multiple conditions and concepts. Overall, the results reflect a well-calibrated policy that adapts reasoning depth based on task complexity.

图 5 给出了一个案例分析，展示模型在 MATH-500 数据集上预测选择 <think> token 的概率。 这一分布表明，模型能够针对不同难度的问题作出平滑且具有层次性的预测。 此外，我们突出展示了分别对应高、中、低置信度的代表性示例，以说明模型的决策行为。 可以看到，被分配到短链推理模式的样本通常是简单的算术题，无需深入或复杂的推理。 相比之下，被路由到思考模式的问题往往更加复杂，涉及多种条件与概念。 总体而言，结果体现了一个校准良好的策略，能够根据任务复杂度调整推理深度。

模型在 MATH-500 上输出 <think> 的概率分布。图中突出显示了概率最高、中等和最低的样本。思考分数接近 0 的示例主要涉及直接计算，而概率为 1.0 的问题更依赖理解和逻辑推理。更多示例与大语言模型回答见附录。

图5：模型在 MATH-500 上输出 <think> 的概率分布。图中突出显示了概率最高、中等和最低的样本。思考分数接近 0 的示例主要涉及直接计算，而概率为 1.0 的问题更依赖理解和逻辑推理。更多示例与大语言模型回答见附录。

5. Limitations and Future Works

This work presents an effective reinforcement learning framework that enables a hybrid model to adapt its inference mode based on both problem complexity and its own capabilities. However, several limitations remain. For instance, during the warm-up phase, we only validate a simple supervised fine-tuning (SFT) approach without extensive parameter tuning to achieve optimal performance, which results in a slight performance drop in the initial model for reinforcement learning. Exploring better strategies for constructing the hybrid model, such as merging techniques or lightweight fine-tuning methods like LoRA to mitigate catastrophic forgetting, could further enhance the overall performance. In addition, while our algorithm has been validated on DeepScaleR, a dataset containing 40K mathematical problems, future work could expand to a broader range of datasets, incorporating more diverse domains to enable more general and practical hybrid reasoning capabilities.

本研究提出了一个有效的强化学习框架，使混合模型能够根据问题复杂度和自身能力调整推理模式。 然而，该框架仍存在一些局限。 例如，在预热阶段，我们只验证了一种简单的有监督微调（SFT）方法，没有通过大量参数调优来达到最优性能，导致用于强化学习的初始模型出现轻微的性能下降。 探索更好的混合模型构建策略，例如模型合并技术，或使用 LoRA 等轻量微调方法缓解灾难性遗忘，可能进一步改善整体性能。 此外，虽然我们的算法已在包含 4 万道数学题的 DeepScaleR 数据集上得到验证，但未来工作可以扩展至更广泛的数据集，涵盖更加多样的领域，从而实现更通用、更实用的混合推理能力。

6. Conclusion

This paper proposes a reinforcement learning framework for building a hybrid reasoning model. It autonomously decides whether to generate a short response or engage in long-form reasoning based on the complexity of the input. The core of our approach is a Decoupled GRPO algorithm, which separates the reinforcement learning objective into two components: mode selection on the control token and accuracy improvement on the response tokens. This decoupling enables a more balanced contribution between the two learning objectives. Our method effectively reduces unnecessary long-form reasoning, thereby lowering overall system cost and improving user latency.

本文提出了一个用于构建混合推理模型的强化学习框架。 它能够根据输入的复杂度，自主决定生成简短回答还是进行长篇推理。 我们方法的核心是解耦 GRPO 算法，将强化学习目标分为两个部分：控制 token 上的模式选择，以及回答 token 上的准确率提升。 这种解耦使两个学习目标的贡献更加均衡。 我们的方法有效减少了不必要的长篇推理，从而降低系统的整体成本，并缩短用户等待时间。