Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
LLMReasoningSurvey香港中文大学澳门大学香港大学普林斯顿大学伊利诺伊大学厄巴纳-香槟分校https://github.com/DevoAllen/Awesome-Reasoning-Economy-Papers
驾驭推理经济性：大语言模型高效推理综述
Abstract
Recent advancements in Large Language Models (LLMs) have significantly enhanced their ability to perform complex reasoning tasks, transitioning from fast and intuitive thinking (System 1) to slow and deep reasoning (System 2). While System 2 reasoning improves task accuracy, it often incurs substantial computational costs due to its slow thinking nature and inefficient or unnecessary reasoning behaviors. In contrast, System 1 reasoning is computationally efficient but leads to suboptimal performance. Consequently, it is critical to balance the trade-off between performance (benefits) and computational costs (budgets), giving rise to the concept of reasoning economy.
大语言模型（LLM）的最新进展显著增强了其执行复杂推理任务的能力，使其从快速、直觉式思考（系统 1）迈向缓慢、深入的推理（系统 2）。 尽管系统 2 推理能够提高任务准确率，但由于其慢思考特性以及低效或不必要的推理行为，它往往会产生巨大的计算成本。 相比之下，系统 1 推理在计算上更高效，但会导致次优性能。 因此，平衡性能（收益）与计算成本（预算）之间的权衡至关重要，由此产生了推理经济性这一概念。
In this survey, we provide a comprehensive analysis of reasoning economy in both the post-training and test-time inference stages of LLMs, encompassing i) the cause of reasoning inefficiency, ii) behavior analysis of different reasoning patterns, and iii) potential solutions to achieve reasoning economy. By offering actionable insights and highlighting open challenges, we aim to shed light on strategies for improving the reasoning economy of LLMs, thereby serving as a valuable resource for advancing research in this evolving area. We also provide a public repository to continually track developments in this fast-evolving field.
在本综述中，我们全面分析 LLM 在后训练阶段和测试时推理阶段的推理经济性，涵盖：（i）推理低效的原因；（ii）不同推理模式的行为分析；以及（iii）实现推理经济性的潜在解决方案。 我们通过提供可操作的见解并强调尚未解决的挑战，力图阐明改善 LLM 推理经济性的策略，从而为推动这一不断演进领域的研究提供宝贵资源。 我们还提供了一个公共仓库，用于持续追踪这一快速发展领域的进展。
Work in progress! Welcome to star and add your papers on the GitHub repo.
本工作仍在进行中！ 欢迎为 GitHub 仓库加星并补充您的论文。
1. Introduction
Large Language Models (LLMs) have demonstrated exceptional performance across various language understanding and generation tasks, particularly with the advent of Chain-of-Thought (CoT) prompting, which encourages models to generate explicit, step-by-step reasoning to arrive at the final answers. While LLMs excel in many scenarios, their reliance on fast, intuitive thinking often falls short when faced with complex reasoning challenges, such as advanced mathematics and coding tasks. Therefore, recent studies try to further boost the reasoning capabilities of LLMs, as exemplified by OpenAI's o1, DeepSeek's R1 and QwQ, to thoroughly explore potential solutions and improve problem-solving accuracy with more slow and deep thinking, leading to the blossoming of Large Reasoning Models (LRMs) and new scaling law during the inference.
大语言模型（LLM）已在各种语言理解与生成任务上展现出卓越性能，尤其是随着思维链（CoT）提示的出现，这种提示鼓励模型生成显式、逐步的推理过程以得到最终答案。 尽管 LLM 在许多场景中表现出色，但面对高等数学和编程任务等复杂推理挑战时，它们所依赖的快速、直觉式思考往往力有不逮。 因此，近期研究试图进一步增强 LLM 的推理能力，OpenAI o1、DeepSeek R1 和 QwQ 便是代表；这些工作通过更缓慢、更深入的思考充分探索潜在解法并提高问题求解准确率，推动了大型推理模型（LRM）的蓬勃发展，也带来了推理阶段的新扩展规律。
While this advancement is remarkable, it comes at a considerable cost. Such LRMs require significantly more time to think before responding, resulting in much longer CoT reasoning and substantial inference overhead. However, not all tasks demand such extensive and deep thinking since tasks vary in complexity, and applying the one-fit-all approach to all tasks results in wasted resources in terms of computation and time. Even worse, not all of the tokens in long thoughts contribute to the final answer. It is found that LRMs often waste excessive resources on unnecessary thoughts (e.g., overthinking) while failing to allocate sufficient computation to truly challenging questions (e.g., underthinking).
尽管这一进展令人瞩目，但它也带来了相当高的代价。 这类 LRM 在作答前需要显著更长的思考时间，因而产生更长的 CoT 推理和巨大的推理开销。 然而，任务复杂度各不相同，并非所有任务都需要如此广泛而深入的思考；对所有任务采用一刀切的方法会浪费计算资源和时间。 更糟的是，长思考中的 token 并非都对最终答案有所贡献。 研究发现，LRM 往往把大量资源浪费在不必要的思考上（例如过度思考），同时又未能为真正具有挑战性的问题分配足够的计算量（例如思考不足）。
This imbalance between capability and efficiency presents a significant challenge - achieving Reasoning Economy, a global optimum that optimizes token usage (budgets) by emphasizing meaningful reasoning steps, reducing redundancy, and dynamically adjusting computational effort based on task complexity. Therefore, it not only ensures the efficiency of LRMs but also unlocks their full potential by "intelligently" stopping or diving deeper like humans (benefits). With the growing importance of the reasoning economy, there is an urgent need to systematically understand and analyze different reasoning behaviors of LRMs, reveal potential challenges toward efficient LRMs, and clearly showcase corresponding solutions to achieve the reasoning economy.
能力与效率之间的这种失衡带来了一项重大挑战，即实现推理经济性：通过强调有意义的推理步骤、减少冗余，并根据任务复杂度动态调整计算投入，优化 token 使用（预算），从而达到全局最优。 因此，它不仅能保证 LRM 的效率，还能让模型像人类一样“智能地”停止思考或进一步深入，从而释放其全部潜力（收益）。 随着推理经济性日益重要，我们迫切需要系统地理解和分析 LRM 的不同推理行为，揭示构建高效 LRM 的潜在挑战，并清晰展示实现推理经济性的相应解决方案。
In this survey, we present the first comprehensive and systematic review of reasoning economy for LRMs. Specifically, we begin by establishing the Foundation of LRMs (Section 2), quickly dissecting how post-training methods (Section 2.1) (i.e, supervised fine-tuning and reinforcement learning) shape reasoning behaviors and how test-time strategies (Section 2.2) (i.e, parallel and sequential) influence the model performance. Building on this foundation, we conduct a rigorous analysis of Challenges towards Reasoning Economy (Section 3), classifying them into inefficient behaviors from the model it self (Section 3.1) or inefficient usage during test-time (Section 3.2).
在本综述中，我们首次对 LRM 的推理经济性进行了全面而系统的回顾。 具体而言，我们首先建立 LRM 的基础（第 2 节），快速剖析后训练方法（第 2.1 节，即监督微调和强化学习）如何塑造推理行为，以及测试时策略（第 2.2 节，即并行方法和顺序方法）如何影响模型性能。 在这一基础之上，我们严格分析推理经济性面临的挑战（第 3 节），并将其分为模型自身的低效行为（第 3.1 节）和测试时的低效使用方式（第 3.2 节）。
Finally, we discuss potential solutions for optimizing reasoning economy in terms of two directions: i) behaviors regulations in post-training (Section 4), which targets undesirable reasoning behaviors at their source in terms of data, algorithm and even model architecture, and ii) usage improvement in test-time (Section 5) to dynamically adapt computation. We also discuss several open challenges, and suggest future research directions (Section 6). By offering a clear and structured roadmap, our work aims to provide actionable insights to guide future research and foster the development of the reasoning economy for more sustainable LRMs.
最后，我们从两个方向讨论优化推理经济性的潜在解决方案：（i）后训练阶段的行为调控（第 4 节），从数据、算法乃至模型架构层面直接治理不理想推理行为的根源；（ii）测试时的使用改进（第 5 节），以动态适配计算量。 我们还讨论了若干开放挑战，并提出未来研究方向（第 6 节）。 本工作通过提供清晰且结构化的路线图，旨在给出可操作的见解以指导未来研究，并推动推理经济性的发展，从而构建更可持续的 LRM。
2. Foundations of LRMs
This section is about the preliminaries of the fundamental methods that underpin the advanced reasoning capabilities of LRMs, including post-training and test-time methods for LRMs. Starting from the pre-trained LLM, post-training methods will shape the capabilities and behaviors of LRMs; and test-time methods will decide in which way the LLMs are used and approach their upper bound by scaling test-time computation.
本节介绍支撑 LRM 高级推理能力的基础方法，包括 LRM 的后训练方法和测试时方法。 从预训练 LLM 出发，后训练方法会塑造 LRM 的能力与行为；测试时方法则决定如何使用 LLM，并通过扩展测试时计算使其逼近能力上限。
2.1 Post-training
Post-training is a crucial phase in the development of LLMs, aimed at enhancing their instruction-following capabilities and aligning these models with human preferences. This alignment is typically achieved through techniques such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) methods, e.g., PPO and GRPO.
后训练是 LLM 开发中的关键阶段，旨在增强模型遵循指令的能力，并使其与人类偏好对齐。 这种对齐通常通过监督微调（SFT）和强化学习（RL）方法实现，例如 PPO 和 GRPO。
Supervised Fine-tuning. SFT plays a crucial role in enhancing the zero-shot multi-task performance of LLMs. By leveraging high-quality task-specific data, SFT refines the model’s ability to generalize across various domains, improving both accuracy and reliability in real-world applications, such as summarization, machine translation, and question answering.
监督微调。 SFT 在增强 LLM 的零样本多任务性能方面发挥着关键作用。 通过利用高质量的特定任务数据，SFT 能够改善模型跨领域泛化的能力，并提高模型在摘要、机器翻译和问答等真实应用中的准确性与可靠性。
Recent studies have proposed leveraging self-improvement methodologies to enhance the reasoning capabilities of models. STaR employs an iterative approach where the large LLM is prompted to generate multiple reasoning chains until the correct solution is obtained, after which the model is fine-tuned on the complete set of successful reasoning trajectories. The latest work Self-Reasoning Language Models (SRLM) further extends to more general instruction-tuning datasets by mixing with few reasoning catalyst data, enabling SRLM to self-refine its own rationales iteratively.
近期研究提出利用自我改进方法来增强模型的推理能力。 STaR 采用一种迭代方法：提示大型 LLM 生成多条推理链，直到得到正确解法，随后使用全部成功推理轨迹对模型进行微调。 最新的自推理语言模型（SRLM）工作进一步扩展到更通用的指令微调数据集，通过混入少量推理催化数据，使 SRLM 能够迭代地自我改进其推理依据。
Though DeepSeek-AI et al. proves that the SFT stage is no longer a must for LRMs by performing RL for pre-trained model, but they also found that SFT contributes to accelerate the training process and achieve better performance for LRMs.
尽管 DeepSeek-AI 等人证明，对预训练模型直接执行 RL 后，SFT 阶段不再是 LRM 的必需环节，但他们也发现 SFT 有助于加速训练过程，并使 LRM 获得更好的性能。
Reinforcement Learning. In the training of LRMs, Reinforcement Learning (RL) plays a key role in enhancing LLMs' reasoning capabilities by providing rewards for both the underlying reasoning process and the final answer, rather than relying exclusively on token-by-token supervision. Moreover, recent studies further indicate that RL-based approaches significantly enhance the generalizability of LLMs, allowing them to perform better in a wider range of tasks and scenarios. These findings underscore the importance of RL in refining model adaptability and improving reasoning capabilities.
强化学习。 在 LRM 的训练中，强化学习（RL）不再完全依赖逐 token 监督，而是同时对内在推理过程和最终答案提供奖励，因此在增强 LLM 推理能力方面发挥着关键作用。 此外，近期研究进一步表明，基于 RL 的方法能够显著增强 LLM 的泛化能力，使其在更广泛的任务和场景中表现得更好。 这些发现凸显了 RL 对改善模型适应性和推理能力的重要性。
The core focus of reinforcement learning currently lies in the design of the reward signals. According to the different granularity of reward signals, there are two major reward models: the Process Reward Model (PRM) and the Outcome Reward Model (ORM). Specifically, the PRM assigns rewards based on intermediate steps within an action sequence rather than solely on the final outcome. PRM enables more fine-grained learning signals, guiding the LLMs toward optimal policies by rewarding beneficial intermediate behaviors.
当前强化学习的核心重点在于奖励信号的设计。 根据奖励信号粒度的不同，主要有两类奖励模型：过程奖励模型（PRM）和结果奖励模型（ORM）。 具体而言，PRM 根据动作序列中的中间步骤分配奖励，而非仅依据最终结果。 PRM 能够提供更细粒度的学习信号，通过奖励有益的中间行为，引导 LLM 走向最优策略。
However, there are several limitations. One the one hand, the training data of PRM is challenging to obtain, as it either requires extensive human annotations or a large amount of sampling, which limits its application. On the other hand, they may also be overly stringent for the reasoning capabilities of LLMs.
然而，PRM 存在若干局限。 一方面，PRM 的训练数据很难获得，因为它需要大量人工标注或大规模采样，这限制了其应用。 另一方面，这类奖励对于 LLM 的推理能力也可能过于严苛。
In contrast, the ORM assigns rewards based on the final outcome of solutions. ORM is widely used in tasks like mathematical reasoning and decision-making, where the final result could be explicitly given and evaluated. Therefore, it is relatively easier to directly assign the reward based on some rule-based method such as format and answer matching, as used in DeepSeek-AI et al. and ReFT. Despite ORM only provides a supervision signal at the outcome level, it still brings exceptional reasoning capabilities since it allows language models to explore reasoning paths without restrictive constraints.
相比之下，ORM 根据解法的最终结果分配奖励。 ORM 广泛用于数学推理和决策等能够明确给出并评估最终结果的任务。 因此，可以较为容易地依据格式匹配和答案匹配等基于规则的方法直接分配奖励，DeepSeek-AI 等人和 ReFT 均采用了这种做法。 尽管 ORM 只在结果层面提供监督信号，但由于它允许语言模型在不受严格约束的情况下探索推理路径，仍能带来卓越的推理能力。
For example, the R1 model exhibits the "Aha" moment, i.e., the emerged self-refine and critique abilities with only rule-based accuracy reward. In summary, both PRM and ORM offer distinct advantages and limitations in training LRMs. More effective reward modeling combining the strengths of PRM and ORM still remains an open question.
例如，R1 模型表现出“顿悟”时刻，即仅依靠基于规则的准确率奖励便涌现出自我改进和批判能力。 总之，PRM 和 ORM 在训练 LRM 时各有优势与局限。 如何构建更有效的奖励模型以结合 PRM 与 ORM 的优势，仍是一个开放问题。
2.2 Test-time Methods
Test-time methods aim to increase the compute for LLMs at test time to get more accurate and reliable results without post-training and often lift the LRMs to compare with further post-training. We classify the test-time methods into Parallel and Sequential methods, following Snell et al. and Muennighoff et al.
测试时方法旨在不进行后训练的情况下，增加 LLM 在测试时的计算量，以获得更准确、更可靠的结果，其效果通常足以与进一步后训练后的 LRM 相比。 我们遵循 Snell 等人和 Muennighoff 等人的划分，将测试时方法分为并行方法和顺序方法。
Parallel Methods. Parallel methods will make LLMs generate several solutions simultaneously, then select the final answer by majority voting, i.e., Self-Consistency, or ORM, i.e., best-of-N and weighted majority voting. These approaches leverage the collective wisdom of multiple solutions to improve accuracy and robustness.
并行方法。 并行方法让 LLM 同时生成多个解法，然后通过多数投票（即自洽性）或 ORM（即 best-of-N 和加权多数投票）选择最终答案。 这些方法利用多个解法的集体智慧来提高准确性和稳健性。
Sequential Methods. Sequential methods, in contrast, involve LLMs iteratively refining their previous steps or answers. This category includes Chain-of-Thought, self-refinement, and search methods such as guided beam search, tree-of-thought, and Monte-Carlo Tree Search (MCTS). The execution of these methods often requires a PRM to determine the most promising branches for exploration, thereby enhancing the efficiency and effectiveness of the solution search process.
顺序方法。 相比之下，顺序方法让 LLM 迭代改进先前的步骤或答案。 这类方法包括思维链、自我改进，以及引导式束搜索、思维树和蒙特卡洛树搜索（MCTS）等搜索方法。 执行这些方法通常需要 PRM 来确定最值得探索的分支，从而提高解法搜索过程的效率与效果。
Summary. Previous work found that the potential of LLMs is not fully reached, and the test-time methods aim to approach the upper bound of LLMs. Brown et al. found that LLaMA-3-8B-Instruct can achieve 98.44% accuracy with 10,000 times repeated sampling and self-consistency, while only 82.9% with 100 samples. The state-of-the-art LRMs, e.g., o1 and R1, all exhibit natural test-time scaling abilities, such as self-refinement, back-tracing, and thought-switching behaviors in their extensive intermediate steps.
总结。 先前工作发现，LLM 的潜力尚未得到充分发挥，而测试时方法旨在 逼近 LLM 的能力上限。 Brown 等人发现，LLaMA-3-8B-Instruct 通过 10,000 次重复采样和自洽性可以达到 98.44% 的准确率，而使用 100 个样本时仅为 82.9%。 最先进的 LRM（例如 o1 和 R1）都在其大量中间步骤中表现出自然的测试时扩展能力，例如自我改进、回溯和思路切换行为。
Moreover, repeated sampling could further improve the performances of R1 and R1-distilled LLMs. In particular, DeepSeek-R1-Distill-Qwen-14B achieves 80% of accuracy on AIME24 by applying majority voting on 64 samples, with only 69.7% accuracy of pass@1. Moreover, Snell et al. found that the test-time methods are even more effective compared with additional training in easy and medium problems, while this is not the case on difficult problems.
此外，重复采样还能进一步提高 R1 和 R1 蒸馏 LLM 的性能。 具体而言，DeepSeek-R1-Distill-Qwen-14B 对 64 个样本应用多数投票后，在 AIME24 上达到 80% 的准确率，而其 pass@1 准确率仅为 69.7%。 此外，Snell 等人发现，对于简单和中等难度问题，测试时方法甚至比额外训练更有效，但在困难问题上并非如此。
3. Challenges Towards Reasoning Economy
In this section, we primarily examine the challenges in LRMs that impair the reasoning economy. We first analyze the Inefficient Model Behaviors caused by post-training methods, which impair the LRMs' performance and lead to computation waste. Then, we exhibit the importance of adaptive test-time setting to save computation while achieving higher performance.
本节主要研究 LRM 中损害推理经济性的挑战。 我们首先分析后训练方法造成的低效模型行为，这些行为会损害 LRM 的性能并导致计算浪费。 随后，我们说明自适应测试时设置对于节省计算并取得更高性能的重要性。
3.1 Inefficient Model Behaviors from Post-training
LRMs are post-trained to align with human preferences and improve reasoning abilities. RL methods play a key role in the post-training stage for LRMs. However, RL optimization relies on reward models (RMs) that are inherently imperfect, primarily due to unreliable human preference annotations. Consequently, over-optimizing based on these flawed RMs can negatively impact the overall capabilities of LLMs.
LRM 通过后训练与人类偏好对齐并提高推理能力。 RL 方法在 LRM 的后训练阶段发挥关键作用。 然而，RL 优化依赖本质上并不完善的奖励模型（RM），这主要源于不可靠的人类偏好标注。 因此，基于这些有缺陷的 RM 进行过度优化，可能会对 LLM 的整体能力产生负面影响。
This situation highlights the risk of reward hacking, where models exploit the reward function to achieve high scores without genuinely aligning with human preferences. As a result, LLMs may exhibit over-optimization or Superficial Alignment, appearing to meet human expectations while lacking true understanding. Next, we will analyze previous research and summarize two notable Superficial Alignment behaviors that directly impact the reasoning efficiency of LLMs: Length Bias and Deceptive Behaviors.
这种情况凸显了奖励黑客的风险，即模型利用奖励函数取得高分，却没有真正与人类偏好对齐。 因此，LLM 可能表现出过度优化或 表面对齐：看似符合人类预期，实际上却缺乏真正的理解。 接下来，我们将分析既有研究，并总结两种直接影响 LLM 推理效率的典型表面对齐行为：长度偏差和欺骗性行为。
3.1.1 Length Bias
One of the most prominent issues arising from Superficial Alignment is Length Bias, where LLMs tend to generate longer responses that contain much redundant content to maximize their reward scores. Previous research has shown that LLMs trained with RL tend to produce longer responses compared to those trained through SFT. As a result, several studies have aimed to answer two key research questions: RQ1: What are the reasons for longer responses? and RQ2: Does the increased length indicate a bias or an enhancement of model capabilities?
表面对齐引发的最突出问题之一是 长度偏差，即 LLM 为最大化奖励分数，倾向于生成包含大量冗余内容的更长回答。 既有研究表明，与通过 SFT 训练的 LLM 相比，使用 RL 训练的 LLM 往往会生成更长的回答。 因此，多项研究试图回答两个关键问题：RQ1：回答变长的原因是什么？ 以及 RQ2：长度增加反映的是偏差，还是模型能力的增强？
Singhal et al. discovered that in existing reward model training datasets, longer responses are often preferred (e.g., RLCD: 63.1%), which leads to a length preference in the RM (RQ1). Hence, the length-biased RM leads the LLMs to generate redundant content with little performance benefits, e.g., too many paraphrased or connection words. Furthermore, they found that using length as a proxy for reward models can yield performance comparable to that of PPO with RMs (RQ2). Other studies have also indicated that RMs struggle to effectively disentangle length bias from response quality during RL training.
Singhal 等人发现，在现有奖励模型训练数据集中，更长的回答往往更受偏好（例如 RLCD 中为 63.1%），从而使 RM 形成长度偏好（RQ1）。 因此，存在长度偏差的 RM 会促使 LLM 生成几乎不带来性能收益的冗余内容，例如过多的同义改写或连接词。 此外，他们发现，以长度作为奖励模型的代理指标，可以取得与使用 RM 的 PPO 相当的性能（RQ2）。 其他研究也指出，在 RL 训练期间，RM 很难有效区分长度偏差与回答质量。
Overly Cautious LRMs. DeepSeek-AI et al. found that an accuracy- and format-based reward function of RL process for LRMs is already effective, and can largely avoid reward hacking. Thus, the Deepseek R1 exhibits a surprising improvement in complex reasoning tasks. Experiments suggest that such improvements from that the LRMs experience an "Aha" moment during the RL process: the reasoning path becomes longer and more complex, leading to behaviors such as self-refinement, recognizing and correcting mistakes, breaking down difficult steps, and iterating on alternative approaches (R1, R2).
过度谨慎的 LRM。 DeepSeek-AI 等人发现，在 LRM 的 RL 过程中，仅使用基于准确率和格式的奖励函数便已有效，并且可以在很大程度上避免奖励黑客。 因此，DeepSeek R1 在复杂推理任务上展现出惊人的提升。 实验表明，这些提升源于 LRM 在 RL 过程中经历了“顿悟”时刻：推理路径变得更长、更复杂，并由此出现自我改进、识别并纠正错误、拆解困难步骤以及迭代尝试替代方法等行为（R1、R2）。
Though progress has been made, recent studies have also identified an "overly cautious" phenomenon in LRMs, characterized by excessive verification and redundant reasoning after giving the right answer (R2). This behavior stems from the assumption that longer outputs are more likely to contain the correct answer or appear more comprehensive, even when shorter, more concise responses would suffice. This overly cautious behavior not only results in inefficient token usage but also hampers LLM performance due to cumulated errors and the "lost in the middle".
尽管已经取得进展，近期研究也在 LRM 中发现了“过度谨慎”现象，其特征是在给出正确答案后仍进行过多验证和冗余推理（R2）。 这种行为源于一种假设：即使更短、更简洁的回答已经足够，更长的输出仍更可能包含正确答案，或显得更加全面。 这种过度谨慎行为不仅导致 token 使用效率低下，还会因错误累积和“中间信息迷失”而损害 LLM 的性能。
Overly Cautious LRMs. LRMs exhibit excessive unnecessary verification and redundant reasoning on easy-to-handle questions or meaningless paraphrases and deviations, leading to inefficient token usage and increased computational costs.
过度谨慎的 LRM。 LRM 会对容易处理的问题进行过量且不必要的验证与冗余推理，或产生毫无意义的同义改写和偏题内容，从而造成 token 使用低效并增加计算成本。
3.1.2 Deceptive Behaviors
Deceptive behavior refers to instances where LLMs appear to align with human preferences, but these behaviors either fail to produce tangible outcomes or conceal other underlying objectives. For example, Greenblatt et al. has demonstrated that LLMs may display differential behaviors across various demographic groups, which could only be found by comparing LLMs' responses to different people. Deceptive behavior is more challenging to detect than length bias or overly cautious behaviors, thus extensive and meticulous human observations are needed to identify it.
欺骗性行为是指 LLM 看似与人类偏好对齐，但这些行为要么无法产生切实结果，要么掩盖了其他潜在目标。 例如，Greenblatt 等人证明，LLM 可能针对不同人口群体表现出差异化行为，而这种现象只有通过比较模型对不同人群的回答才能发现。 欺骗性行为比长度偏差或过度谨慎行为更难检测，因此需要广泛而细致的人类观察才能识别。
Fake Thinking LRMs. In the context of LRMs, recent studies have uncovered a Fake thinking behavior: They tend to generate plausible, sound reasoning steps that lack logical rigor or correctness. Some studies found that LRMs appear to engage in self-refinement or deliberate reasoning processes by analyzing the generated solutions. However, empirical evidence suggests that such behaviors are often merely superficial, with little substantial progress being made toward problem-solving.
虚假思考的 LRM。 在 LRM 场景中，近期研究发现了一种 虚假思考 行为：模型倾向于生成看似合理、通顺，却缺乏逻辑严谨性或正确性的推理步骤。 一些研究发现，LRM 看似会通过分析已生成的解法来进行自我改进或审慎推理。 然而，实证证据表明，这些行为往往仅流于表面，对解决问题几乎没有实质性推进。
These deceptive behaviors can make LLMs seem like they are working, but in fact, they are wasting computational resources, and even hurting their performances. For instance, Wang et al. has found that LRMs tend to abandon high-quality reasoning steps early on and engage in ineffective self-refinement and thought-changing. Anderson et al. found that R1 lacks of confidence and tends to give up quickly on those problems they can explain why the answer is correct and get stuck "thinking forever". According to the reasoning boundary theory, fake thinking can also be treated as the meaningless attempts of LLMs while solving problems beyond their capabilities.
这些欺骗性行为会让 LLM 看起来正在认真工作，但实际上是在浪费计算资源，甚至损害自身性能。 例如，Wang 等人发现，LRM 往往会过早放弃高质量推理步骤，转而进行无效的自我改进和思路切换。 Anderson 等人发现，R1 缺乏信心，面对那些它能够解释答案为何正确的问题时也倾向于迅速放弃，并陷入“永远思考”的状态。 根据推理边界理论，虚假思考也可以视为 LLM 在解决超出自身能力的问题时所作的无意义尝试。
Fake Thinking LRMs. LRMs appear to work towards problem-solving, but these actions are often superficial and do not lead to meaningful progress.
虚假思考的 LRM。 LRM 看似在努力解决问题，但这些行动通常流于表面，无法带来有意义的进展。
3.2 Inefficient Model Usage in Test-time
Though the test-time methods could push the performance of LRMs further in a training-free way, the application of test-time methods is often suboptimal. Previous work found that two dimensions significantly influence the test-time performance of LLMs: the selection of inference algorithm and test-time computation allocated to each question.
尽管测试时方法可以在无需训练的情况下进一步提高 LRM 的性能，但测试时方法的应用往往并非最优。 先前工作发现，两个维度会显著影响 LLM 的测试时性能：推理算法的选择，以及分配给每个问题的测试时计算量。
3.2.1 Unreasonable Algorithm Selection
To employ a test-time method, there are two dimensions to be decided if we do not need to consider computation limitation: (1) which algorithm and (2) what are the hyper-parameters. The often case of employing the test-time method is to select one (e.g., sampling) and set its parameters (e.g., temperature, top-p). However, these are not sufficient to achieve efficient reasoning.
在不考虑计算限制的情况下，采用测试时方法需要决定两个方面：（1）使用哪种算法；（2）采用哪些超参数。 常见做法是选择一种测试时方法（例如采样），并设置其参数（例如温度和 top-p）。 然而，仅做到这些还不足以实现高效推理。
For the choice of inference algorithm, Parashar et al. found that there is not a single inference algorithm that suits all of the tasks. To be more specific, Chen et al. found that majority voting in LLMs improves accuracy on simple problems but degrades performance on complex ones as votes increase. Similarly, Snell et al. concluded that search-based methods outperform parallel methods on harder problems. Setlur et al. argue that the verifier is a must for test-time scaling. As for the parameter setting of the inference algorithm, recent work found that a lower temperature of sampling is more suitable for reasoning tasks, while creative tasks need a higher temperature.
在推理算法的选择方面，Parashar 等人发现，不存在一种适合所有任务的推理算法。 更具体地说，Chen 等人发现，随着投票次数增加，多数投票会提高 LLM 在简单问题上的准确率，却会降低其在复杂问题上的性能。 类似地，Snell 等人得出结论：在较困难的问题上，基于搜索的方法优于并行方法。 Setlur 等人认为，验证器是测试时扩展的必要组成部分。 在推理算法的参数设置方面，近期工作发现，较低的采样温度更适合推理任务，而创造性任务需要更高的温度。
The above studies emphasize the need for algorithmic adaptability based on task complexity.
上述研究强调了 根据任务复杂度调整算法的必要性。
3.2.2 Unreasonable Computation Allocation
As mentioned before, though scaling the computation could bring consistent performance benefits, scaling LLaMA-3-8B-Instruct from generating 100 samples to 10,000 samples is often unacceptable for a simple question. However, a more complex problem is worth a high computation budget, while a small computation budget could lead to suboptimal accuracy. Recent studies propose the Reasoning Boundary of LRMs, in which they found that middle-complexity problems need more computation.
如前所述，尽管扩展计算量可以持续带来性能收益，但对于一个简单问题，让 LLaMA-3-8B-Instruct 的生成样本数从 100 个增加到 10,000 个通常是不可接受的。 然而，更复杂的问题值得更高的计算预算，而较小的计算预算可能导致次优准确率。 近期研究提出 LRM 的推理边界，并发现中等复杂度的问题需要更多计算量。
For the sequential inference algorithm, recent studies found that longer solutions with more self-refinement are not necessarily better. Specifically, Wu et al. found that there is an optimal length for sequential refinement, and harder problems need a longer optimal length. Snell et al. made an empirical study and found that there is an optimal computation allocation for problems with different complexities. Recently, Yang et al. found that there exists an optimal length distribution across different domains. Hence, inadequate computation will lead to the wrong answer, while more tokens impair the performance.
对于顺序推理算法，近期研究发现，包含更多自我改进的更长解法不一定更好。 具体而言，Wu 等人发现，顺序改进存在一个最优长度，而且越困难的问题需要越长的最优长度。 Snell 等人通过实证研究发现，不同复杂度的问题各自存在最优的计算分配。 近期，Yang 等人发现，不同领域之间存在最优的长度分布。 因此，计算不足会导致错误答案，而更多 token 也可能损害性能。
Above all, previous work empirically found that there is not a one-for-all computation setting for all tasks and samples varied in complexity. They emphasize the importance of adaptive computation allocation based on task complexity.
综上，既有工作通过实证发现，不存在一种适用于所有任务和不同复杂度样本的统一计算设置。 这些工作强调了 根据任务复杂度自适应分配计算量的重要性。
4. Optimization for Reasoning Economy — part-1: Post-training
In this section, we provide an overview of post-training optimization methods aimed at enhancing reasoning economy in LLMs. These methods focus on regulating LLM behaviors to improve efficiency and reduce unnecessary computational overhead from Data, Algorithm, and Architecture perspectives.
本节概述旨在增强 LLM 推理经济性的后训练优化方法。 这些方法从数据、算法和架构三个角度调控 LLM 行为，以提高效率并减少不必要的计算开销。
4.1 Data
High Quality Data Construction. By explicitly encoding desired reasoning patterns and behaviors, researchers can guide LLMs toward more advanced and effective performance. Qin et al. utilize small-scale long-thought datasets sampled by test-time scaling to enhance the reasoning performance of LLMs, and find that the obtained LLM exhibits clear long thought reasoning patterns. Similarly, Muennighoff et al. demonstrated that just 1,000 high-quality and diverse samples for SFT could produce competitive LRMs comparable to advanced models like o1-preview. They found that the Quality, Diversity, and Difficulty of constructed data are important.
高质量数据构建。 通过显式编码期望的推理模式与行为，研究者可以引导 LLM 获得更先进、更有效的性能。 Qin 等人利用测试时扩展采样得到的小规模长思考数据集来增强 LLM 的推理性能，并发现所得 LLM 呈现出明显的长思考推理模式。 类似地，Muennighoff 等人证明，仅使用 1,000 个高质量且多样化的样本进行 SFT，就能得到可与 o1-preview 等先进模型竞争的 LRM。 他们发现，构建数据的质量、多样性和难度十分重要。
4.2 Algorithm
Recent research has focused on three key strategies: optimizing reward structures to balance response length and quality (Long2short RL), enforcing explicit length constraints during training and inference (Adaptive Budget-aware Tuning), and developing compression techniques to eliminate redundant reasoning steps while preserving accuracy (CoT Compression). These approaches collectively aim to enhance inference efficiency by either directly guiding models to generate concise responses or restructuring reasoning processes for more compact representations.
近期研究聚焦三项关键策略：优化奖励结构以平衡回答长度与质量（Long2short RL）、在训练和推理期间施加显式长度约束（自适应预算感知调优），以及开发在保持准确率的同时消除冗余推理步骤的压缩技术（CoT 压缩）。 这些方法通过直接引导模型生成简洁回答，或重构推理过程以获得更紧凑的表示，共同提升推理效率。
4.2.1 Long2short RL
To address the inefficiencies caused by length bias in RL-tuned LLMs, researchers have proposed various reward design improvements. For example, Singhal et al. explored intuitive approaches, such as increasing the KL coefficient, applying length penalties to reward model scores, and discarding overly long rollouts. Despite these efforts, RL-tuned models still tend to produce longer responses than SFT models.
为解决 RL 调优 LLM 中长度偏差造成的低效问题，研究者提出了多种奖励设计改进方案。 例如，Singhal 等人探索了若干直观方法，包括增大 KL 系数、对奖励模型分数施加长度惩罚，以及丢弃过长的 rollout。 尽管做出了这些努力，RL 调优模型仍倾向于生成比 SFT 模型更长的回答。
Recently, Kimi Team et al. proposed the long2short RL approach, which aims to transform lengthy and unnecessary reasoning processes into concise and accurate ones. They investigated strategies like model merging, shortest rejection sampling, and DPO optimization for shorter responses. They also introduced long2short RL, which uses a normalized reward model across multiple responses, significantly reducing output length while maintaining reasoning quality. The following explores the long2short RL approach designed to improve the efficiency and effectiveness of RL-tuned LLMs.
近期，Kimi Team 等人提出 long2short RL，旨在把冗长且不必要的推理过程转化为简洁、准确的过程。 他们研究了模型合并、最短拒绝采样以及面向更短回答的 DPO 优化等策略。 他们还提出 long2short RL，在多个回答之间使用归一化奖励模型，从而在保持推理质量的同时显著缩短输出长度。 下面将探讨旨在提高 RL 调优 LLM 效率与有效性的 long2short RL 方法。
Quality-Length Reward Disentangle. One approach to mitigate length bias is to develop more sophisticated reward models that can better distinguish between response quality and length. Chen et al. and Park et al. jointly train two reward heads on shared feature representations, one trained to correlate with length, and the other trained to focus on quality while ignoring the length.
质量与长度奖励解耦。 缓解长度偏差的一种方法，是开发能够更好地区分回答质量与长度的复杂奖励模型。 Chen 等人和 Park 等人在共享特征表示上联合训练两个奖励头，其中一个学习与长度相关，另一个则学习聚焦质量并忽略长度。
Length Penalty or Normalization. Meng et al. proposed a simple length normalization way for DPO to eliminate the influence of length, which has been proven quite effective in alleviating the length bias. Yeo et al. utilize a cosine reward to incentivize different length scaling behaviors, eliminating length bias. Chen et al. observed that LRMs often provide multiple correct answers within a single solution. Leveraging this pattern, they split these solutions into multiple shorter ones and constructed preference pairs, with the first short and correct answer as the preferred solution. They optimized the LLMs using DPO and SimPO, finding SimPO to be more effective, reducing response lengths by 30% to 40%.
长度惩罚或归一化。 Meng 等人为 DPO 提出了一种简单的长度归一化方法，以消除长度影响；事实证明，该方法能相当有效地缓解长度偏差。 Yeo 等人使用余弦奖励来激励不同的长度扩展行为，从而消除长度偏差。 Chen 等人观察到，LRM 经常在单个解法中给出多个正确答案。 他们利用这一模式，把这些解法拆分成多个较短解法并构造偏好对，将第一个简短且正确的答案作为偏好解法。 他们使用 DPO 和 SimPO 优化 LLM，并发现 SimPO 更为有效，可将回答长度缩短 30% 至 40%。
Procedure Reward. Rule-based reward, i.e., accuracy-based reward, has been proved to be quite effective, and less likely to suffer from reward hacking. However, a recent study highlights a limitation of accuracy-based rewards in shaping LLM reasoning: They fail to ensure efficient utilization of scaled computation during inference. To address this, the authors propose the Meta Reinforcement Finetuning (MRT) by using cumulative regret as a reinforcement learning objective, encouraging LLMs to make incremental progress with each token. This approach was shown to enable stable performance improvements as computational resources scale.
过程奖励。 基于规则的奖励，即基于准确率的奖励，已被证明十分有效，而且不易遭受奖励黑客。 然而，近期一项研究指出，基于准确率的奖励在塑造 LLM 推理方面存在局限：它无法确保在推理过程中高效利用扩展后的计算量。 为解决这一问题，我们提出元强化微调（MRT），将累积遗憾作为强化学习目标，鼓励 LLM 随每个 token 逐步取得进展。 研究表明，随着计算资源扩展，该方法能够带来稳定的性能提升。
4.2.2 Adaptive Budget-aware Tuning
Some studies have focused on explicitly guiding LLMs to adhere to token budgets by specifying desired response lengths in prompts. More importantly, they further post-trained LRMs to follow budget constraints better while achieving higher performance. Yuan et al. trained LLMs to follow length constraints by constructing SFT data with explicit length specifications. Aggarwal and Welleck extended this approach by using RL to optimize models for both accuracy and length control. Their results demonstrated superior performance across various token budget settings.
一些研究通过在提示中指定期望的回答长度，显式引导 LLM 遵守 token 预算。 更重要的是，这些研究进一步对 LRM 进行后训练，使其在取得更高性能的同时更好地遵循预算约束。 Yuan 等人通过构建带有显式长度要求的 SFT 数据，训练 LLM 遵循长度约束。 Aggarwal 和 Welleck 使用 RL 同时针对准确率与长度控制优化模型，从而扩展了这一方法。 他们的结果表明，该方法在多种 token 预算设置下均表现出更优性能。
Additionally, it has been observed that there is a token elasticity phenomenon, meaning that overly strict constraints can lead to increased token costs. To address this issue, a budget prediction and allocation paradigm was implemented. Specifically, a zero-shot or regression-based budget estimator was used to predict a suitable budget, thereby avoiding excessive computation and overly strict constraints. This approach achieved a 67% reduction in response length with only a 3% loss in accuracy.
此外，研究还观察到 token 弹性现象，即过于严格的约束反而会增加 token 成本。 为解决这一问题，研究者采用了预算预测与分配范式。 具体而言，该方法使用零样本或基于回归的预算估计器预测合适预算，从而避免过度计算和过于严格的约束。 该方法在准确率仅损失 3% 的情况下，将回答长度缩短了 67%。
4.2.3 CoT Compression
It is evident that longer CoT suffers from token redundancy and inference latency, which is not desired in practice. Thus, several studies try to identify important tokens and eliminate unnecessary tokens or reasoning steps, therefore enhancing the inference economy. These methods can be broadly categorized into two categories: 1) explicit compression that directly enforces the model to generate more concise reasoning by fine-tuning on carefully curated datasets or providing specific demonstrations.; and 2) implicit compression that maps multiple reasoning tokens or steps into a continuous space to achieve a more compact representation.
显然，更长的 CoT 会带来 token 冗余和推理延迟，这在实际应用中并不理想。 因此，多项研究尝试识别重要 token，并消除不必要的 token 或推理步骤，从而增强推理经济性。 这些方法大致可分为两类：（1）显式压缩，通过在精心整理的数据集上微调或提供特定示例，直接迫使模型生成更简洁的推理；（2）隐式压缩，将多个推理 token 或步骤映射到连续空间，以获得更紧凑的表示。
Explicit Compression. The first step of compression is to identify key tokens or step in the reasoning processing, while most methods rely on perplexity as a primary metric, using it as a proxy to determine the importance of individual tokens or steps. Some of the methods refine the supervised fine-tuning dataset, replacing lengthy reasoning with identified key reasoning steps as target outputs. Another approach uses key reasoning steps as demonstrations for in-context learning, guiding LLMs to generate only essential steps during inference while minimizing unimportant tokens. A key limitation of these methods is the potential loss of coherence and a comprehensive understanding of the full reasoning process. While the full CoT can be reconstructed using additional LLMs, the recovered version may not fully align with the original, leading to inconsistencies or missing nuances and additional inference costs.
显式压缩。 压缩的第一步是识别推理过程中的关键 token 或步骤，而多数方法依赖困惑度这一主要指标，并将其作为判断各 token 或步骤重要性的代理。 一些方法改进监督微调数据集，用识别出的关键推理步骤替代冗长推理，作为目标输出。 另一种方法把关键推理步骤用作上下文学习示例，引导 LLM 在推理时仅生成必要步骤，同时尽量减少不重要的 token。 这些方法的一项关键局限，是可能损失连贯性以及对完整推理过程的全面理解。 尽管可以使用额外 LLM 重构完整 CoT，但恢复后的版本可能无法与原始过程完全一致，从而产生不一致、遗漏细微信息，并增加推理成本。
Implicit Compression. Besides explicit compression, another line of work argues that the explicit language space (i.e., tokens) may not always be optimal for reasoning, as most word tokens are primarily for textual coherence and not essential for reasoning. The earlier work focuses on the implicit chain-of-thought approach, which compiles explicit CoT reasoning of the teacher model into a student model that directly produces the final answer to develop more efficient reasoning. Another representative work is Coconut (Chain of Continuous Thought), which feed hidden state instead of specific token back to the LLM as the subsequent input embedding, allowing the model to encode multiple alternative next reasoning steps in the continuous space and thus reducing the token cost, followed by Cheng et al. Recent advances in memory-efficient architectures, such as Anchor-based LLMs (AnLLMs), further highlight the inefficiency of token-based representations by demonstrating that compressing sequence information into anchor tokens can significantly reduce memory usage while maintaining accuracy, followed by Zhang et al. To reduce the need to fully finetune the models, recent studies use an additional projection module to inject the compressed continuous states of the source model into the target model.
隐式压缩。 除显式压缩外，另一类工作认为，显式语言空间（即 token）并非总是最适合推理，因为大多数词 token 主要用于维持文本连贯性，而不是推理所必需的。 早期工作聚焦隐式思维链方法，将教师模型的显式 CoT 推理编译进直接生成最终答案的学生模型，从而形成更高效的推理。 另一项代表性工作是 Coconut（连续思维链）：它把隐藏状态而非特定 token 反馈给 LLM，作为后续输入嵌入，使模型能够在连续空间中编码多个候选的下一推理步骤，从而降低 token 成本；Cheng 等人随后延续了这一方向。 Anchor-based LLM（AnLLM）等内存高效架构的最新进展进一步揭示了基于 token 的表示效率低下：把序列信息压缩进锚点 token，可以在保持准确率的同时显著减少内存使用；Zhang 等人随后延续了这一方向。 为减少完整微调模型的需求，近期研究使用额外的投影模块，将源模型压缩后的连续状态注入目标模型。
4.3 Architecture
One approach to enhancing efficiency is reducing active model parameters, requiring adjustments in model or system architecture. Two key strategies are integrating System-1 and System-2 thinking and adaptively utilizing model depth. System-1 and System-2 cooperation enables dynamic selection between fast, intuitive reasoning and slower, deliberate processing, optimizing efficiency. Meanwhile, adaptive parameter activation optimizes model depth and resource allocation during inference, balancing performance and computational cost.
提升效率的一种方法是减少活跃模型参数，这需要调整模型或系统架构。 两项关键策略是整合系统 1 与系统 2 思考，以及自适应利用模型深度。 系统 1 与系统 2 的协作能够在快速、直觉式推理和较慢、审慎的处理之间动态选择，从而优化效率。 与此同时，自适应参数激活会在推理期间优化模型深度和资源分配，以平衡性能与计算成本。
4.3.1 System-1 and System-2 Cooperation
Drawing inspiration from human cognition, several studies propose different methods to dynamically switch from slow, conscious, deliberate reasoning (System 2) to fast, automatic, intuitive thinking (System 1), therefore achieving a better balance between fast and accurate thought. Specifically, there are different paths towards this goal.
受人类认知启发，多项研究提出了不同方法，在缓慢、有意识、审慎的推理（系统 2）与快速、自动、直觉式思考（系统 1）之间动态切换，从而更好地平衡思考速度与准确性。 具体而言，实现这一目标存在多条路径。
Single-Model Routing. It is straightforward to empower one model with abilities to switch between fast (system-1) and slow (system-2) inference methods according to difficulty signals, thereby optimizing both efficiency and effectiveness. On the one hand, the choice can be left to the user to determine the extent of cognitive effort exerted by the model. For instance, OpenAI has equipped the O1 with three distinct thinking modes: low, middle, and high. The high thinking mode delivers accurate yet lengthy responses tailored for extremely challenging problems, while the low mode provides swift responses that are satisfactory.
单模型路由。 一种直接做法，是赋予单个模型根据难度信号在快速（系统 1）和缓慢（系统 2）推理方法之间切换的能力，从而同时优化效率和效果。 一方面，可以把选择权交给用户，由用户决定模型投入多少认知努力。 例如，OpenAI 为 O1 提供了三种不同的思考模式：低、中和高。 高思考模式针对极具挑战性的问题给出准确但较长的回答，而低模式则快速给出令人满意的回答。
However, this direction has not been sufficiently explored thus far. How to train models to possess diverse modes of thinking and how to evaluate the capability gap between different modes of thinking in models remain underexplored areas.
然而，这一方向迄今尚未得到充分探索。 如何训练模型具备多样化的思考模式，以及如何评估模型不同思考模式之间的能力差距，仍是研究不足的领域。
On the other hand, the model can be designed to autonomously determine the appropriate level of cognitive effort based on the complexity of the task at hand. This autonomous selection process can be facilitated by incorporating a mechanism that assesses the difficulty of the input and dynamically adjusts the inference method accordingly. Ding et al. leverages a binary router to assign queries to smaller or larger based on the difficulty and expected quality. Pan et al. use consistency verification and reasoning steps as complexity signals to determine which questions require slow, deliberate thinking and which do not. Similarly, Saha et al. curates a dataset that trains models to decompose problems into sub-goals of varying difficulty levels and adaptively apply different strategies to solve them.
另一方面，可以让模型根据当前任务的复杂度，自主确定适当的认知努力水平。 可以引入一种评估输入难度并据此动态调整推理方法的机制，来促进这一自主选择过程。 Ding 等人利用二元路由器，根据难度和预期质量把查询分配给较小或较大的模型。 Pan 等人使用一致性验证和推理步数作为复杂度信号，以确定哪些问题需要缓慢、审慎的思考，哪些不需要。 类似地，Saha 等人整理了一个数据集，用于训练模型把问题分解为不同难度的子目标，并自适应地采用不同策略求解。
Multi-Model Collaboration. One common approach to accelerating inference is speculative decoding, which follows a draft-then-verify paradigm. This method first generates multiple token candidates efficiently and then verifies them in parallel. Xia et al. formally analyze this method through a formal study and extensive discussion of both drafting and verification phases, while Kim et al. propose the BiLD framework, where a smaller model generates text at low cost, and a larger model refines errors in parallel. Further research explores improved drafting and verification strategies using specific reward model, with a broader overview in Xia et al.
多模型协作。 加速推理的一种常见方法是推测解码，它遵循“先起草、后验证”的范式。 该方法首先高效生成多个候选 token，然后并行验证它们。 Xia 等人通过形式化研究以及对起草和验证两个阶段的广泛讨论，对该方法进行了正式分析；Kim 等人则提出 BiLD 框架，由较小模型低成本生成文本，再由较大模型并行修正错误。 进一步研究使用特定奖励模型探索了更好的起草与验证策略，Xia 等人对此提供了更广泛的综述。
In addition, some methods take an agentic approach, dynamically determining when to engage larger models for refinement at different granularity. Specifically, SwiftSage invokes system 2 thinking for subgoals during inference, while CITER focuses on token-level refinement when needed.
此外，一些方法采用智能体式方案，动态决定何时让更大模型以不同粒度进行改进。 具体而言，SwiftSage 在推理期间为子目标调用系统 2 思考，而 CITER 则在需要时聚焦 token 级改进。
Knowledge Distillation. Another approach to achieving a balance between slow and fast reasoning is knowledge distillation, where knowledge from a larger, more complex model (System 2) is transferred to a smaller, more efficient model (System 1). The distilled smaller model will be capable of advanced reasoning abilities while costing less in computations. In detail, Yu et al. leverages several self-supervised methods to distill higher-quality outputs from System 2 techniques back into LLM generations without intermediate reasoning token sequences. More recently, it has been proved that direct distillation from DeepSeek-R1 outperforms applying RL on the base model such as Qwen2.5-32B, setting a new record on the reasoning benchmarks. There are also emerging studies exploring distillation from Transformer to lower computation complexity architectures, such as Mamba, achieving much faster inference for large batches and long sequences with the Mamba-based model.
知识蒸馏。 在缓慢推理与快速推理之间取得平衡的另一种方法是知识蒸馏，即把较大、更复杂模型（系统 2）的知识迁移到较小、更高效的模型（系统 1）。 蒸馏后的小模型能够具备高级推理能力，同时消耗更少计算量。 具体而言，Yu 等人利用多种自监督方法，将系统 2 技术产生的更高质量输出蒸馏回不含中间推理 token 序列的 LLM 生成过程。 近期研究还证明，直接从 DeepSeek-R1 蒸馏优于在 Qwen2.5-32B 等基础模型上应用 RL，并在推理基准上创下新纪录。 也有新兴研究探索从 Transformer 向 Mamba 等计算复杂度更低的架构进行蒸馏，使基于 Mamba 的模型在大批量和长序列场景中实现快得多的推理。
4.3.2 Adaptive Activated Parameters
Performance improvements of deep learning models largely benefit from advancements in deep network design. Present LLMs also pay great attention to deep architecture designs. For models with comparable scales of parameters, increasing model depths often yields more significant benefits than broadening widths, especially in mathematical reasoning tasks. However, deepening layers will result in larger model sizes, which means that relatively small models with few layers are hard to resolve complex reasoning tasks. Additionally, increasing model layers leads to a linear increase in inference computation costs. From the perspective of model depth, we can enhance the inference capabilities of relatively small-scale LLMs by recurrently utilizing intermediate layers or accelerate the inference of deep LLMs by skipping some intermediate layers.
深度学习模型的性能提升很大程度上受益于深层网络设计的进步。 当前 LLM 也高度重视深层架构设计。 对于参数规模相近的模型，增加模型深度往往比扩展宽度带来更显著的收益，尤其是在数学推理任务中。 然而，加深网络层数会导致模型规模变大，这意味着层数较少的相对小型模型很难解决复杂推理任务。 此外，增加模型层数会使推理计算成本线性增长。 从模型深度角度看，我们可以通过循环使用中间层来增强相对小规模 LLM 的推理能力，也可以通过跳过部分中间层来加速深层 LLM 的推理。
Recurrent Layers. Recurrent mechanisms enable LLMs to perform arbitrarily many computations before emitting a token, which is a simple solution for test-time compute scaling for relatively small LLMs with few layers. Recent work has described depth-recurrent, looped Transformers and studied their potential benefits with careful theoretical and small-scale analysis. Specifically, Fan et al. established the superior length generalization capabilities of looped Transformers. Yu et al. proposes a framework enhancing CoT reasoning by combining looped and auto-regressive Transformers. Recurrently utilizing layers in Transformer-based LLMs enhances reasoning ability on small-scale LLMs.
循环层。 循环机制使 LLM 能够在输出一个 token 前执行任意多次计算，为层数较少的相对小型 LLM 提供了一种简单的测试时计算扩展方案。 近期工作描述了深度循环的循环式 Transformer，并通过细致的理论分析和小规模分析研究其潜在收益。 具体而言，Fan 等人证实了循环式 Transformer 卓越的长度泛化能力。 Yu 等人提出一个结合循环式 Transformer 与自回归 Transformer 的框架，以增强 CoT 推理。 在基于 Transformer 的 LLM 中循环使用网络层，可以增强小规模 LLM 的推理能力。
Dynamic Depth. Additionally, inspired by efficient reasoning using model pruning and sparse models, recent works validate that not all layers of LLMs are necessary during inference. In this way, two methods involving dynamic depths of LLMs are included: Early Exit and Skip Layer. Since simpler tasks require fewer layers, while more complex tasks require more layers of inference, achieving efficient LLM inference then becomes when to stop the inference process adaptively based on the input instance.
动态深度。 此外，受利用模型剪枝和稀疏模型实现高效推理的启发，近期工作验证了 LLM 并非所有网络层在推理期间都是必需的。 由此产生了两种涉及 LLM 动态深度的方法：早退和跳层。 简单任务需要较少网络层，而复杂任务需要更多推理层，因此实现高效 LLM 推理的关键变成了如何根据输入实例自适应地停止推理过程。
5. Optimization for Reasoning Economy — part-2: Test-time Methods
Test-time methods can enhance the performance of LRMs without the need for tuning. However, as previously noted (Section 3.2), there is significant potential for optimizing test-time usage to achieve greater reasoning efficiency. In this section, we survey existing methods aiming for reasoning economy by optimizing test-time methods to more economically allocating inference computation as shown in Figure 图5.
测试时方法无需调优即可增强 LRM 的性能。 然而，如第 3.2 节所述，优化测试时使用方式以获得更高推理效率仍具有巨大潜力。 本节综述通过优化测试时方法、更经济地分配推理计算来实现推理经济性的既有工作，如 图5 所示。
5.1 Input-side Optimization
In this section, we introduce those works that attempt to optimize the usage of LLMs by controlling model behaviors through explicit constraints in the input.
本节介绍通过在输入中施加显式约束来控制模型行为、进而优化 LLM 使用方式的工作。
5.1.1 Adaptive Budget Allocation before Decoding
The main idea of Adaptive Budget Allocation before Decoding is to perform Computation Budget Prediction & Allocation before Decoding, i.e., first decide a computation budget for the question, and force the LLMs to follow the constraint.
解码前自适应预算分配的核心思想，是在解码前执行计算预算预测与分配，即先为问题确定计算预算，再迫使 LLM 遵守该约束。
Budget Prediction. The basic idea of predicting the budget is to consider the difficulty of the problems to the reasoning LLM. According to previous works, the confidence of the model to solve the question can be estimated and predicted. As for the computation prediction methods for each question, Han et al. collected a batch of questions with their optimal budgets and trained a regression model to predict the computation needed for novel prompts. Meanwhile, Wang et al. considered the difficulty of the questions and allocated budgets similar to those of questions with comparable difficulty.
预算预测。 预算预测的基本思想，是考虑推理 LLM 所面对问题的难度。 根据先前工作，可以估计并预测模型解决问题的置信度。 在逐问题计算量预测方面，Han 等人收集了一批带有最优预算的问题，并训练回归模型来预测新提示所需的计算量。 与此同时，Wang 等人考虑问题难度，并为其分配与相近难度问题类似的预算。
Budget Constrained Generation. With the budget constraint, then we can indicate the length constraint in the prompt, thereby instructing the LLMs to give the response while satisfying the constraint. Previous work found that LLMs possess the capacities to follow the length constraint to some extent. To further improve token budget awareness of LLMs, Han et al. also post-trained LLMs to better follow length constraints.
预算约束生成。 有了预算约束后，我们可以在提示中指出长度约束，从而要求 LLM 在满足约束的同时给出回答。 先前工作发现，LLM 在一定程度上具备遵循长度约束的能力。 为进一步提高 LLM 的 token 预算意识，Han 等人还对 LLM 进行后训练，使其更好地遵循长度约束。
5.2 Output-side Optimization
In this section, we introduce the work of optimizing the usage of LLMs at the output end of the model. This includes Constrained Decoding, Adaptive Algorithm Selection, and Adaptive Computation Allocation during Decoding.
本节介绍在模型输出端优化 LLM 使用方式的工作。 其中包括约束解码、自适应算法选择和解码期间的自适应计算分配。
5.2.1 Adaptive Algorithm Selection
The exploration of adaptive choice of test-time algorithms is quite limited, especially on efficient thinking. Here, we present several works that adaptively adjust the parameters of test-time algorithms, which could be potentially used to achieve algorithm determination.
关于自适应选择测试时算法的探索仍然十分有限，尤其是在高效思考方面。 这里，我们介绍若干自适应调整测试时算法参数的工作，这些工作有望用于实现算法选择。
Snell et al. demonstrates the potential of adaptive method selection, showing that optimal settings can achieve up to 4x greater efficiency compared to the PRM best-of-N approach. This underscores the benefits of dynamically tailoring test-time strategies to specific tasks. Dhuliawala et al. introduces an Adaptive Decoding Layer combined with Latent Preference Optimization (LPO), enabling fine-grained temperature prediction at the token or sample level for more precise generation control. Chen et al. propose using high initial temperatures to enhance response diversity, which expands the search space and increases the likelihood of discovering correct answers.
Snell 等人展示了自适应方法选择的潜力：与 PRM best-of-N 方法相比，最优设置最高可实现 4 倍效率提升。 这凸显了针对具体任务动态定制测试时策略的益处。 Dhuliawala 等人提出将自适应解码层与潜在偏好优化（LPO）相结合，在 token 级或样本级进行细粒度温度预测，从而实现更精确的生成控制。 Chen 等人提出使用较高的初始温度来增强回答多样性，从而扩大搜索空间并提高发现正确答案的可能性。
These approaches collectively emphasize the importance of adaptive mechanisms in optimizing the choice of test-time algorithms, paving the way for more efficient and effective reasoning in LLMs.
这些方法共同强调了自适应机制在优化测试时算法选择中的重要性，为 LLM 更高效、更有效的推理铺平了道路。
5.2.2 Adaptive Budget Allocation During Decoding
Early Stopping. The first line of adaptive budget allocation is early stopping. Manvi et al. utilizes the self-evaluation abilities of LLMs to decide whether the current solution is good enough during the sequentially self-refinement procedure. Aggarwal et al. argue that a constant sample number for self-consistency is not cost-efficient, and proposed to stop sampling when a consistency rate is reached. Li et al. splits the whole sampling window for self-consistency into several smaller ones and stops sampling if all of the answers within a small window are the same, thereby greatly reducing the computation cost without performance sacrifice.
提前停止。 自适应预算分配的第一条路线是提前停止。 Manvi 等人利用 LLM 的自我评估能力，在顺序自我改进过程中判断当前解法是否已经足够好。 Aggarwal 等人认为，为自洽性使用固定样本数并不具备成本效益，因此提出在达到一定一致率时停止采样。 Li 等人把自洽性的完整采样窗口拆分为多个较小窗口，并在某个小窗口内所有答案都相同时停止采样，从而在不牺牲性能的前提下大幅降低计算成本。
Recent studies argue that not all solutions should be treated equally when determining when to perform early stopping. Wan et al. takes into account both the quality of the reasoning path and whether a high level of consistency has been achieved. Huang et al., on the other hand, utilizes the model's confidence to select the preferred reasoning path and combines this with consistency judgment to decide whether to stop early.
近期研究认为，在判断何时提前停止时，不应平等对待所有解法。 Wan 等人同时考虑推理路径的质量以及是否已经达到较高的一致性水平。 另一方面，Huang 等人利用模型置信度选择偏好的推理路径，并将其与一致性判断结合，以决定是否提前停止。
Search with Pruning. The second line is pruning while searching, no matter the parallel or sequential methods. The goal is to prune low-quality search branches early while retaining high-quality ones, thereby saving computational resources. For sequential test-time methods, RMs are often used to guide the search procedure by pruning low-quality branches and rolling out high-quality ones. Guided search could eliminate unnecessary computation usage while improving the accuracy of LRMs.
带剪枝的搜索。 第二条路线是在搜索时进行剪枝，无论采用并行方法还是顺序方法。 其目标是在保留高质量分支的同时尽早剪除低质量搜索分支，从而节省计算资源。 对于顺序测试时方法，通常使用 RM 引导搜索过程：剪除低质量分支，并继续展开高质量分支。 引导式搜索可以在提高 LRM 准确率的同时，消除不必要的计算消耗。
As for parallel test-time methods, Wang et al. utilizes the self-evaluation abilities of LLMs to prune the low-quality samples during best-of-N sampling. To expand the search space of best-of-N, Sun et al. set a very large sample window at the beginning of the sampling, large enough to consume all of the GPU memory, then gradually pruned those low-quality and completed the saved samples. Zhu et al. select the top high-quality steps and use these steps to guide and constrain the LLMs to complete these prefixes.
对于并行测试时方法，Wang 等人在 best-of-N 采样过程中利用 LLM 的自我评估能力剪除低质量样本。 为扩大 best-of-N 的搜索空间，Sun 等人在采样开始时设置一个足以占满全部 GPU 内存的超大采样窗口，随后逐步剪除低质量样本，并完成保留下来的样本。 Zhu 等人选择质量最高的步骤，并使用这些步骤引导和约束 LLM 完成相应前缀。
Constrained Decoding. Identifying suboptimal thinking patterns is often challenging, which makes mitigating them particularly difficult. Hence, previous works have to utilize human analysis to diagnose the deceptive behaviors of LLMs and more importantly, to utilize human observation to inspire specific solutions for them.
约束解码。 识别次优思考模式通常很有挑战性，因此缓解这些模式尤其困难。 因此，先前工作不得不利用人工分析诊断 LLM 的欺骗性行为；更重要的是，还要利用人工观察启发针对这些行为的具体解决方案。
For example, for fake self-refine of LLMs, Huang et al. and Gou et al. suggest using external feedback to assist the self-refine procedure of LLMs. Other works utilize the human-observed behaviors to design force decoding paradigm, aiming to solve the fake thinking and overly cautious behaviors of LRMs, e.g., enforcing LLMs to adhere to the previous thought or performing self-refine at proper time. Wang et al. found that the proposed Thought Switching Penalty approach can encourage LLMs to perform deeper reasoning and reduce unreasonable thought-switching.
例如，针对 LLM 的虚假自我改进，Huang 等人和 Gou 等人建议使用外部反馈辅助 LLM 的自我改进过程。 其他工作利用人工观察到的行为设计强制解码范式，旨在解决 LRM 的虚假思考和过度谨慎行为，例如强制 LLM 坚持先前思路，或在适当时机执行自我改进。 Wang 等人发现，所提出的思路切换惩罚方法可以鼓励 LLM 进行更深入的推理，并减少不合理的思路切换。
6. Discussion
Efficient Multi-modal Reasoning. Multi-modal large language models (MLLMs) have demonstrated promising capabilities in various multi-modal reasoning tasks, including but not limited to mathematical reasoning, visual question answering (VQA), and multi-modal dialogue systems. Recent advances, exemplified by o1-like and R1-like technologies, have further catalyzed progress in this field. Current approaches on efficient multi-modal reasoning primarily focus on improvements within MLLMs themselves, mainly including: i) model architecture optimization (e.g., lightweight vision encoders, vision token compression, vision-language projector, small language models), and efficient structures; ii) efficient vision techniques adoption (e.g. Pruning, knowledge distillation, and ViT quantization), among other methods.
高效多模态推理。 多模态大语言模型（MLLM）已在多种多模态推理任务中展现出可观能力，包括但不限于数学推理、视觉问答（VQA）和多模态对话系统。 以类 o1 和类 R1 技术为代表的最新进展，进一步推动了这一领域的发展。 当前高效多模态推理方法主要聚焦于改进 MLLM 本身，主要包括：（i）模型架构优化，例如轻量级视觉编码器、视觉 token 压缩、视觉语言投影器、小语言模型以及高效结构；（ii）采用高效视觉技术，例如剪枝、知识蒸馏和 ViT 量化等。
However, the evaluation and targeted optimization of efficiency in multi-modal (long-) reasoning remain relatively preliminary. Jiang et al. proposes two metrics for measuring the efficiency of multi-model CoT: relevance rate, which calculates the proportion of content contributing to answering, and reflection quality, which evaluates whether reflection steps can effectively correct previous mistakes or present new insights. Xiang et al. proposes a self-structured COT paradigm that decomposes reasoning into minimal semantic atomic steps to achieve efficient multi-modal reasoning by generating dynamic structures and lengths based on the problem types. Moreover, most of the aforementioned efficient reasoning methods for LLMs can also be effectively applied to MLLMs.
然而，多模态（长）推理的效率评估与针对性优化仍处于相对初步的阶段。 Jiang 等人提出两个衡量多模态 CoT 效率的指标：相关率用于计算对回答有贡献的内容占比，反思质量则评估反思步骤能否有效纠正先前错误或提出新见解。 Xiang 等人提出自结构化 COT 范式，把推理分解为最小语义原子步骤，并根据问题类型生成动态结构和长度，从而实现高效多模态推理。 此外，前述大多数面向 LLM 的高效推理方法也可以有效应用于 MLLM。
Efficient Agentic Reasoning. The advent of LRMs has also brought substantial performance improvements to AI agents. Using RL training similar to o1, the OpenAI's Deep Research utilizes advanced reasoning capabilities to synthesize large amounts of online information, attaining notable performance on the challenging Humanity's Last Exam. Recent research has increasingly focused on leveraging long reasoning capabilities in agent systems, where cutting-edge implementations integrate these with retrieval, tool augmentation, domain knowledge incorporation, and other auxiliary components to push performance boundaries.
高效智能体推理。 LRM 的出现也为 AI 智能体带来了显著的性能提升。 OpenAI 的 Deep Research 使用类似 o1 的 RL 训练，利用高级推理能力综合大量在线信息，并在极具挑战性的 Humanity's Last Exam 上取得显著表现。 近期研究日益关注在智能体系统中利用长推理能力，前沿实现把它与检索、工具增强、领域知识引入及其他辅助组件相结合，以不断突破性能边界。
To analyze the limited effectiveness of LRMs in interactive environments, Cuadron et al. presents the first comprehensive empirical study in agentic tasks, analyzes the overthinking in LRMs, including its manifestations, quantification, and impact on different models, and also proposes potential mitigation approaches accordingly. Zhou et al. proposes the LaRMA framework to explore the necessity of reasoning capabilities in LRMs for agent scenarios, analyzes the performance differences between LRMs and LLMs across various tasks, and provides insights into optimizing agent performance through hybrid LLM-LRM configurations. Some studies also investigate efficient agent training methods and efficient interaction schemes for multi-agent systems.
为分析 LRM 在交互式环境中效果有限的原因，Cuadron 等人首次对智能体任务进行了全面实证研究，分析 LRM 的过度思考现象，包括其表现形式、量化方式以及对不同模型的影响，并据此提出潜在缓解方法。 Zhou 等人提出 LaRMA 框架，探索智能体场景中 LRM 推理能力的必要性，分析 LRM 与 LLM 在多种任务上的性能差异，并为通过混合 LLM-LRM 配置优化智能体性能提供见解。 一些研究还探讨了高效智能体训练方法，以及多智能体系统的高效交互方案。
Evaluation Metrics and Benchmarks. With the proliferation of long reasoning and the increasing severity of the over-reasoning problem, researchers have recently begun developing specialized benchmarks and metrics to quantitatively measure reasoning efficiency. Hashemi et al. proposes DNA Bench to expose the vulnerability of current LRMs and reveals that LRMs generate up to 70× more unnecessary tokens and fail at tasks that non-reasoning models handle efficiently. Chen et al. introduces outcome-based and process-based efficiency metrics and concludes that LRMs overthink on simple problems, with later solutions contributing little to accuracy and diversity. Conversely, Wang et al. introduces an underthinking metric to reveal the underthinking problem in o1-like LLMs, where models frequently switch wrong reasoning thoughts without fully exploring potentially correct initial thoughts, leading to insufficient reasoning depth and poor performance. Anderson et al. creates a dataset from NPR Sunday Puzzle Challenges to evaluate LRM's reasoning with general knowledge (not PhD-level), also revealing behaviors similar to overthinking.
评估指标与基准。 随着长推理日益普及、过度推理问题愈发严重，研究者近期开始开发专门的基准和指标，以定量衡量推理效率。 Hashemi 等人提出 DNA Bench 来揭示当前 LRM 的脆弱性，并发现 LRM 会生成多达 70 倍的不必要 token，而且在非推理模型能够高效处理的任务上失败。 Chen 等人提出基于结果和基于过程的效率指标，并得出结论：LRM 会在简单问题上过度思考，后续解法对准确率和多样性的贡献很小。 相反，Wang 等人提出思考不足指标，以揭示类 o1 LLM 的思考不足问题：模型频繁切换到错误的推理思路，却没有充分探索可能正确的初始思路，因而导致推理深度不足和性能不佳。 Anderson 等人基于 NPR Sunday Puzzle Challenges 构建数据集，用于评估 LRM 运用常识知识（而非博士级知识）进行推理的能力，同时也揭示了类似过度思考的行为。
Explainability of LRMs. The study of the explainability of black-box LLMs has always been a topic of interest. Particularly, LRMs have explored on their own through RL and have demonstrated reasoning abilities on par with those of human PhD students. However, the mechanisms underlying their achievement of such performance remain enigmatic. Current research on LRMs often focuses on their behavior analysis, such as the observation of overly cautious or fake thinking behaviors, and then retraces the post-training algorithms or test-time methods.
LRM 的可解释性。 黑盒 LLM 的可解释性研究一直备受关注。 尤其是，LRM 通过 RL 自主探索，并展现出可与人类博士生相媲美的推理能力。 然而，它们取得这种性能的底层机制仍然成谜。 当前关于 LRM 的研究往往聚焦行为分析，例如观察过度谨慎或虚假思考行为，再回溯后训练算法或测试时方法。
Nevertheless, it is essential to focus on how these models work on the inside, probing the internal mechanisms of LRMs. Moreover, more user-friendly and powerful toolkits are also critical for research purposes and large-scale human analysis. This would help us understand the thought patterns of LRMs, identify their faults, and provide directions for further improvement.
尽管如此，关注这些模型内部如何工作并探查 LRM 的内部机制仍至关重要。 此外，更加易用且强大的工具包对于研究和大规模人工分析也十分关键。 这将帮助我们理解 LRM 的思考模式、识别其缺陷，并为进一步改进提供方向。
7. Conclusion
In this survey, we systematically examined the challenges and solutions involved in achieving an inference economy for large reasoning models, emphasizing the urgent need for efficient reasoning mechanisms that balance computational cost with performance. To the best of our knowledge, this is the first comprehensive review that analyzes the underlying causes, observed phenomena, key challenges, and emerging solutions for enabling efficient reasoning in LLMs, offering a structured roadmap and actionable strategies for practical deployment. By anchoring progress in the principles of inference economy, this survey serves not only as a synthesis of current knowledge but also as a call to action for further research in this path, highlighting the importance of developing more sustainable and scalable models that reason not only effectively but also efficiently.
在本综述中，我们系统考察了大型推理模型实现推理经济性所涉及的挑战与解决方案，并强调迫切需要在计算成本与性能之间取得平衡的高效推理机制。 据我们所知，这是首篇全面分析 LLM 高效推理的根本原因、观察现象、关键挑战与新兴解决方案的综述，并为实际部署提供结构化路线图和可操作策略。 本综述以推理经济性原则为进展基础，不仅综合了现有知识，也呼吁沿这一方向开展进一步研究，强调开发更可持续、更可扩展模型的重要性，使模型的推理不仅有效，而且高效。