---

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

LLMReasoningTest-Time Compute加州大学伯克利分校Google DeepMind

Snell C, Lee J, Xu K, et al. Scaling LLM test-time compute optimally can be more effective than scaling model parameters[PP/OL]. V1. arXiv (2024-08-06)[2026-11-04]. DOI:10.48550/arXiv.2408.03314.

https://arxiv.org/abs/2408.03314v1

---

最优扩展大语言模型的测试时计算，可能比扩大模型参数更有效

Abstract

Enabling LLMs to improve their outputs by using more test-time computation is a critical step towards building generally self-improving agents that can operate on open-ended natural language. In this paper, we study the scaling of inference-time computation in LLMs, with a focus on answering the question: if an LLM is allowed to use a fixed but non-trivial amount of inference-time compute, how much can it improve its performance on a challenging prompt? Answering this question has implications not only on the achievable performance of LLMs, but also on the future of LLM pretraining and how one should tradeoff inference-time and pre-training compute. Despite its importance, little research attempted to understand the scaling behaviors of various test-time inference methods. Moreover, current work largely provides negative results for a number of these strategies.

让大语言模型（LLM）通过使用更多测试时计算来改进输出，是构建能够处理开放式自然语言、具备通用自我改进能力的智能体的关键一步。 本文研究 LLM 中推理时计算的扩展，重点回答以下问题：如果允许 LLM 使用固定但不可忽略的一定量推理计算，它在具有挑战性的提示上的表现能够提升多少？ 这个问题的答案不仅影响 LLM 能达到的性能，也关系到 LLM 预训练的未来，以及应当如何权衡推理时计算与预训练计算。 尽管这一问题十分重要，但鲜有研究尝试理解各种测试时推理方法的扩展行为。 此外，现有工作对其中若干策略主要给出了负面的结果。

In this work, we analyze two primary mechanisms to scale test-time computation: (1) searching against dense, process-based verifier reward models; and (2) updating the model's distribution over a response adaptively, given the prompt at test time. We find that in both cases, the effectiveness of different approaches to scaling test-time compute critically varies depending on the difficulty of the prompt. This observation motivates applying a “compute-optimal” scaling strategy, which acts to most effectively allocate test-time compute adaptively per prompt. Using this compute-optimal strategy, we can improve the efficiency of test-time compute scaling by more than 4$\times$ compared to a best-of-N baseline. Additionally, in a FLOPs-matched evaluation, we find that on problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14$\times$ larger model.

在本研究中，我们分析两种扩展测试时计算的主要机制：（1） 依据稠密的、基于过程的验证器奖励模型进行搜索；（2） 在测试时根据给定提示，自适应地更新模型的回答分布。 我们发现，在这两种情况下，不同测试时计算扩展方法的有效性都高度依赖于提示的难度。 这一观察促使我们采用一种“计算最优”的扩展策略，根据每个提示自适应地分配测试时计算，使其发挥最大效用。 使用这一计算最优策略，相比 best-of-N 基线，我们能够将测试时计算扩展的效率提升至 4 倍以上。 此外，在 FLOPs 匹配的评估中，我们发现，对于较小基础模型已经能够取得一定成功率的问题，测试时计算可以使其超过参数规模为其 14 倍 的模型。

1. Introduction

Humans tend to think for longer on difficult problems to reliably improve their decisions. Can we instill a similar capability into today's large language models (LLMs)? More specifically, given a challenging input query, can we enable language models to most effectively make use of additional computation at test time so as to improve the accuracy of their response? In theory, by applying additional computation at test time, an LLM should be able to do better than what it was trained to do.

人类在面对困难问题时往往会思考更长时间，从而可靠地改善决策。 我们能否让当今的大语言模型（LLM）也具备类似能力？ 更具体地说，给定一个具有挑战性的输入查询，我们能否让语言模型最有效地利用测试时的额外计算，从而提高回答的准确率？ 理论上，通过在测试时投入额外计算，LLM 应当能够超越其训练所赋予的表现。

In addition, such a capability at test-time also has the potential to unlock new avenues in agentic and reasoning tasks. For instance, if pre-trained model size can be traded off for additional computation during inference, this would enable LLM deployment in use-cases where smaller on-device models could be used in place of datacenter scale LLMs. Automating the generation of improved model outputs by using additional inference-time computation also provides a path towards a general self-improvement algorithm that can function with reduced human supervision.

此外，这种测试时能力还有望为智能体任务和推理任务开辟新的途径。 例如，如果可以用推理阶段的额外计算来换取更小的预训练模型，就能在一些应用场景中部署较小的端侧模型，替代数据中心规模的 LLM。 利用额外推理时计算，自动生成质量更高的模型输出，也为构建一种能够减少人工监督的通用自我改进算法提供了路径。

图1：主要结果概览。左：迭代自我改进（即修订）与搜索的计算最优扩展。左侧比较了 PaLM 2-S* 修订模型的计算最优扩展策略与修订设置（上）和 PRM 搜索设置（下）的基线。在修订设置中，标准 best-of-N（例如“并行”）与计算最优扩展之间的差距逐渐扩大，使后者仅用 1/4 的测试时计算便能超过 best-of-N。同样，在 PRM 搜索中，计算最优扩展在早期便显著优于 best-of-N，并且在部分位置，使用其 1/4 的计算量时已几乎能超过 best-of-N。详见第 5、6 节。右：比较测试时计算扩展与模型参数扩展。我们比较 PaLM 2-S* 的计算最优测试时扩展，与一个参数规模约为其 14 倍、但不投入额外测试时计算（例如采用贪心采样）的预训练模型。假设两个模型都使用 X 个预训练 token 和 Y 个推理 token，训练更大的模型会同时成倍增加这两项 FLOPs 需求。如果给较小模型投入额外测试时计算，使其 FLOPs 需求与较大模型相同，其准确率会如何？对于修订（上），当 Y ≪ X 时，测试时计算往往优于额外预训练。随着推理与预训练 token 数之比增大，测试时计算在简单问题上仍更有优势，而在较难问题上预训练更优。PRM 搜索（下）也呈现类似趋势。详见第 7 节。

Prior work studying inference-time computation provides mixed results. On the one hand, some works show that current LLMs can use test-time computation to improve their outputs, on the other hand, other work shows that the effectiveness of these methods on more complex tasks such as math reasoning remains highly limited, even though reasoning problems often require drawing inferences about existing knowledge as opposed to new knowledge. These sorts of conflicting findings motivate the need for a systematic analysis of different approaches for scaling test-time compute.

已有的推理时计算研究得出了不一致的结果。 一方面，一些工作表明当前的 LLM 能够利用测试时计算改进输出；另一方面，另一些工作表明，这些方法在数学推理等更复杂任务上的有效性仍然十分有限，尽管推理问题往往要求根据已有知识进行推断，而非获取新知识。 这些相互矛盾的发现说明，有必要系统地分析不同的测试时计算扩展方法。

We are interested in understanding the benefits of scaling up test-time compute. Arguably the simplest and most well-studied approach for scaling test-time computation is best-of-N sampling: sampling N outputs in “parallel” from a base LLM and selecting the one that scores the highest per a learned verifier or a reward model. However, this approach is not the only way to use test-time compute to improve LLMs. By modifying either the proposal distribution from which responses are obtained (for instance, by asking the base model to revise its original responses “sequentially”) or by altering how the verifier is used (e.g. by training a process-based dense verifier and searching against this verifier), the ability scale test-time compute could be greatly improved, as we show in the paper.

我们希望理解增加测试时计算带来的收益。 可以说，最简单、研究最充分的测试时计算扩展方法是 best-of-N 采样：从基础 LLM 中“并行”采样 N 个输出，再选择经学习得到的验证器或奖励模型评分最高的那个。 然而，这并非利用测试时计算改善 LLM 的唯一方式。 正如本文所展示的，通过修改生成回答所依据的 提议分布（例如，让基础模型“顺序地”修订其原始回答），或改变 验证器 的使用方式（例如，训练基于过程的稠密验证器，并依据该验证器进行搜索），可以显著增强测试时计算的扩展能力。

To understand the benefits of scaling up test-time computation, we carry out experiments on the challenging MATH benchmark using PaLM-2 models specifically fine-tuned to either revise incorrect answers (e.g. improving the proposal distribution; Section 6) or verify the correctness of individual steps in an answer using a process-based reward model (PRM) (Section 5). With both approaches, we find that the efficacy of a particular test-time compute strategy depends critically on both the nature of the specific problem at hand and the base LLM used. For example, on easier problems, for which the base LLM can already readily produce reasonable responses, allowing the model to iteratively refine its initial answer by predicting a sequence of N revisions (i.e., modifying the proposal distribution), may be a more effective use of test-time compute than sampling N independent responses in parallel. On the other hand, with more difficult problems that may require searching over many different high-level approaches to solving the problem, re-sampling new responses independently in parallel or deploying tree-search against a process-based reward model is likely a more effective way to use test-time computation.

为了理解增加测试时计算的收益，我们在具有挑战性的 MATH 基准上开展实验，使用经过专门微调的 PaLM-2 模型，使其能够修订错误回答（例如，改进提议分布；见 第 6 节），或使用过程奖励模型（PRM）验证回答中各个步骤的正确性（见 第 5 节）。 在这两种方法中，我们都发现，特定测试时计算策略的效果高度依赖于当前问题的具体性质，以及所使用的基础 LLM。 例如，对于较容易的问题，基础 LLM 已经能够较为轻松地产生合理回答；此时，让模型预测一系列共 N 次修订，以迭代改进初始答案（即修改提议分布），可能比并行采样 N 个独立回答更有效地利用测试时计算。 另一方面，对于可能需要搜索多种高层解题思路的更困难问题，重新独立地并行采样新回答，或依据过程奖励模型进行树搜索，可能是更有效的测试时计算方式。

This finding illustrates the need to deploy an adaptive “compute-optimal” strategy for scaling test-time compute, wherein the specific approach for utilizing test-time compute is selected depending on the prompt, so as to make the best use of additional computation. We also show that a notion of question difficulty (Section 4) from the perspective of the base LLM can be used to predict the efficacy of test-time computation, enabling us to practically instantiate this ‘compute-optimal’ strategy given a prompt. By appropriately allocating test-time compute in this way, we are able to greatly improve test-time compute scaling, surpassing the performance of a best-of-N baseline while only using about 4x less computation with both revisions and search (Sections 5 and 6).

这一发现说明，有必要采用自适应的“计算最优”策略来扩展测试时计算，根据提示选择具体的测试时计算方法，以充分利用额外计算。 我们还表明，从基础 LLM 的角度定义的 问题难度（见 第 4 节）可以用来预测测试时计算的有效性，使我们能够针对给定提示实际构造这一“计算最优”策略。 通过这种方式合理分配测试时计算，我们能够显著改善其扩展效果：对于修订和搜索，两者都只需约 1/4 的计算量，就能超过 best-of-N 基线的表现（见 第 5 节 和 第 6 节）。

Using our improved test-time compute scaling strategy, we then aim to understand to what extent test-time computation can effectively substitute for additional pretraining. We conduct a FLOPs-matched comparison between a smaller model with additional test-time compute and pretraining a 14x larger model. We find that on easy and intermediate questions, and even hard questions (depending on the specific conditions on the pretraining and inference workload), additional test-time compute is often preferable to scaling pretraining. This finding suggests that rather than focusing purely on scaling pretraining, in some settings it is be more effective to pretrain smaller models with less compute, and then apply test-time compute to improve model outputs.

利用改进后的测试时计算扩展策略，我们进一步研究测试时计算能够在多大程度上有效替代额外的预训练。 我们在 FLOPs 匹配的条件下，对投入额外测试时计算的较小模型，与预训练一个 参数规模大 14 倍的模型 进行比较。 我们发现，对于简单和中等难度的问题，乃至困难问题（取决于预训练与推理工作负载的具体条件），增加测试时计算往往优于扩大预训练。 这一发现表明，除了纯粹扩大预训练之外，在某些情形下，用较少计算预训练更小的模型，再通过测试时计算改进模型输出，会更有效。

That said, with the most challenging questions, we observe very little benefits from scaling up test-time compute. Instead, we find that on these questions, it is more effective to make progress by applying additional pretraining compute, demonstrating that current approaches to scaling test-time compute may not be 1-to-1 exchangeable with scaling pretraining. Overall, this suggests that even with a fairly naïve methodology, scaling up test-time computation can already serve to be more preferable to scaling up pretraining, with only more improvements to be attained as test-time strategies mature. Longer term, this hints at a future where fewer FLOPs are spent during pretraining and more FLOPs are spent at inference.

不过，对于最具挑战性的问题，我们观察到增加测试时计算带来的收益很小。 相反，我们发现，在这些问题上增加预训练计算能够更有效地取得进展，这说明当前的测试时计算扩展方法可能无法与扩大预训练进行一比一替换。 总体而言，这表明，即使采用相当朴素的方法，增加测试时计算也已经可能优于扩大预训练；随着测试时策略逐渐成熟，还将获得更多改进。 从长远来看，这预示着未来可能在预训练阶段投入更少 FLOPs，而在推理阶段投入更多 FLOPs。

2. A Unified Perspective on Test-Time Computation: Proposer and Verifier

We first unify approaches for using test-time computation and then analyze some representative methods. First, we view the use of additional test-time compute through the lens of modifying the model's predicted distribution adaptively at test-time, conditioned on a given prompt. Ideally, test-time compute should modify the distribution so as to generate better outputs than naïvely sampling from the LLM itself would. In general, there are two knobs to induce modifications to an LLM's distribution: (1) at the input level: by augmenting the given prompt with an additional set of tokens that the LLM conditions on to obtain the modified distribution, or (2) at the output level: by sampling multiple candidates from the standard LM and performing surgery on these candidates.

我们首先统一使用测试时计算的各种方法，再分析其中一些有代表性的方法。 首先，我们将额外测试时计算视为：在测试时，以给定提示为条件，自适应地 修改模型的预测分布。 理想情况下，测试时计算应当通过修改分布，生成比直接从 LLM 本身朴素采样更好的输出。 一般而言，有两个调节方向可以改变 LLM 的分布：（1）在输入层面，为给定提示补充一组额外 token，让 LLM 以这些 token 为条件得到修改后的分布；或 （2）在输出层面，从标准语言模型中采样多个候选，并对这些候选进行处理。

In other words, we could either modify the proposal distribution induced by the LLM itself such that it is an improvement over naïvely conditioning on the prompt or we could use some post-hoc verifiers or scorers to perform output modifications. This process is reminiscent of Markov chain Monte Carlo (MCMC) sampling from a complex target distribution but by combining a simple proposal distribution and a score function. Modifying the proposal distribution directly by altering input tokens and using a verifier form two independent axes of our study.

换句话说，我们可以修改 LLM 本身诱导的 提议分布，使其优于仅以提示为条件的朴素分布；也可以使用某些事后的 验证器或评分器 来修改输出。 这一过程让人联想到马尔可夫链蒙特卡洛（MCMC）：通过组合简单的提议分布与评分函数，从复杂的目标分布中进行采样。 通过改变输入 token 直接修改提议分布，以及使用验证器，构成了本研究的两个独立维度。

Modifying the proposal distribution. One way to improve the proposal distribution is to directly optimize the model for a given reasoning task via RL-inspired finetuning methods such as STaR or ${\text{ReST}}^{\text{EM}}$. Note that these techniques do not utilize any additional input tokens but specifically finetune the model to induce an improved proposal distribution. Instead, techniques such as self-critique enable the model itself to improve its own proposal distribution at test time by instructing it to critique and revise its own outputs in an iterative fashion. Since prompting off-the-shelf models is not effective at enabling effective revisions at test time, we specifically finetune models to iteratively revise their answers in complex reasoning-based settings. To do so, we utilize the approach of finetuning on on-policy data with Best-of-N guided improvements to the model response.

修改提议分布。 改进提议分布的一种方式，是采用受强化学习启发的微调方法，例如 STaR 或 ${\text{ReST}}^{\text{EM}}$，针对给定推理任务直接优化模型。 注意，这些技术不使用任何额外的输入 token，而是专门对模型进行微调，从而得到改进后的提议分布。 相比之下，自我批评等技术通过指示模型迭代地批评和修订自身输出，使模型能够在测试时自行改进提议分布。 由于对现成模型进行提示并不能让其在测试时有效修订，我们专门微调模型，使其能够在复杂的推理场景中迭代修订答案。 为此，我们在同策略数据上进行微调，并使用 Best-of-N 引导模型回答的改进。

Optimizing the verifier. In our abstraction of the proposal distribution and verifier, the verifier is used to aggregate or select the best answer from the proposal distribution. The most canonical way to use such a verifier is by applying best-of-N sampling, wherein we sample N complete solutions and then select the best one according to a verifier. However, this approach can be further improved by training a process-based verifier, or a process reward model (PRM), which produces a prediction of the correctness of each intermediate step in an solution, rather than just the final answer. We can then utilize these per-step predictions to perform tree search over the space of solutions, enabling a potentially more efficient and effective way to search against a verifier, compared to naïve best-of-N.

优化验证器。 在我们对提议分布与验证器的抽象中，验证器用于聚合提议分布中的答案，或从中选择最佳答案。 使用这种验证器最典型的方式是 best-of-N 采样，即采样 N 个完整解答，然后根据验证器选择最佳解答。 不过，通过训练基于过程的验证器，即过程奖励模型（PRM），还可以进一步改进这一方法：PRM 预测解答中每个中间步骤的正确性，而非只预测最终答案。 随后，我们可以利用这些逐步预测，在解答空间中进行树搜索；相比朴素的 best-of-N，这可能是一种更高效、更有效的验证器搜索方式。

3. How to Scale Test-Time Computation Optimally

Given the unification of various methods, we would now like to understand how to most effectively utilize test-time computation to improve LM performance on a given prompt. Concretely we wish to answer:

在统一各种方法之后，我们希望理解如何 最有效地 利用测试时计算，改善语言模型在给定提示上的表现。 具体而言，我们希望回答：

Problem setup

We are given a prompt and a test-time compute budget within which to solve the problem. Under the abstraction above, there are different ways to utilize test-time computation. Each of these methods may be more or less effective depending on the specific problem given. How can we determine the most effective way to utilize test-time compute for a given prompt? And how well would this do against simply utilizing a much bigger pretrained model?

问题设置

给定一个提示，以及用于解决该问题的测试时计算预算。 在上述抽象下，存在不同的测试时计算使用方式。 每种方法的有效程度可能因具体问题而异。 对于给定提示，我们如何确定利用测试时计算的 最有效 方式？ 与直接使用一个大得多的预训练模型相比，这种方式能取得怎样的效果？

When either refining the proposal distribution or searching against a verifier, there are several different hyper-parameters that can be adjusted to determine how a test-time compute budget should be allocated. For example, when using a model finetuned for revisions as the proposal distribution and an ORM as the verifier, we could either spend the full test-time compute budget on generating N independent samples in parallel from the model and then apply best-of-N, or we could sample N revisions in sequence using a revision model and then select the best answer in the sequence with an ORM, or strike a balance between these extremes. Intuitively, we might expect “easier” problems to benefit more from revisions, since the model's initial samples are more likely to be on the right track but may just need further refinement. On the other hand, challenging problems may require more exploration of different high-level problem solving strategies, so sampling many times independently in parallel may be preferable in this setting.

无论是改进提议分布，还是依据验证器进行搜索，都有多个超参数可以调节，以决定如何分配测试时计算预算。 例如，当使用为修订而微调的模型作为提议分布，并使用结果奖励模型（ORM）作为验证器时，我们既可以将全部测试时计算预算用于从模型中并行生成 N 个独立样本，再应用 best-of-N；也可以使用修订模型依次采样 N 次修订，然后用 ORM 选出序列中的最佳答案；还可以在这两种极端之间取得平衡。 直观上，我们可能预期“较容易”的问题从修订中获益更多，因为模型的初始样本更可能已经沿着正确方向，只需进一步改进。 另一方面，具有挑战性的问题可能需要探索更多不同的高层解题策略，因此在这种情况下，多次独立的并行采样可能更合适。

In the case of verifiers, we also have the option to choose between different search algorithms (e.g. beam-search, lookahead-search, best-of-N), each of which may exhibit different properties depending on the quality of the verifier and proposal distribution at hand. More sophisticated search procedures might be more useful in harder problems compared to a much simpler best-of-N or majority baseline.

对于验证器，我们还可以在不同搜索算法之间选择，例如束搜索、前瞻搜索和 best-of-N；它们的性质可能因当前验证器与提议分布的质量而异。 相较于简单得多的 best-of-N 或多数投票基线，更复杂的搜索过程可能在较难问题上更有用。

3.1. Test-Time Compute-Optimal Scaling Strategy

In general, we would therefore like to select the optimal allocation of our test-time compute budget for a given problem. To this end, for any given approach of utilizing test-time compute (e.g., revisions and search against a verifier in this paper, various other methods elsewhere), we define the “test-time compute-optimal scaling strategy” as the strategy that chooses hyperparameters corresponding to a given test-time strategy for maximal performance benefits on a given prompt at test time. Formally, define $\mathrm{Target}(\theta ,N,q)$ as the distribution over natural language output tokens induced by the model for a given prompt $q$, using test-time compute hyper-parameters $\theta$, and a compute budget of $N$. We would like to select the hyper-parameters $\theta$ which maximize the accuracy of the target distribution for a given problem. We express this formally as:

因此，一般来说，我们希望针对给定问题，为测试时计算预算选择 最优 的分配方案。 为此，对于任何使用测试时计算的方法（例如本文中的修订和依据验证器进行搜索，以及其他研究中的各种方法），我们将 “测试时计算最优扩展策略” 定义为：在测试时，针对给定提示选择某一测试时策略的相应超参数，以获得最大性能收益的策略。 形式上，定义 $\mathrm{Target}(\theta ,N,q)$ 为模型针对给定提示 $q$，在采用测试时计算超参数 $\theta$ 和计算预算 $N$ 时，诱导出的自然语言输出 token 分布。 我们希望选择超参数 $\theta$，使目标分布在给定问题上的准确率最大化。 形式化地表示为：

$$\begin{array}{}\text{(1)}& \begin{array}{rl}& {\theta }_{q,a^{\ast }(q)}^{\ast }(N)=\underset{\theta }{\mathrm{argmax}}\\ & \left({\mathbb{E}}_{y\sim \mathrm{Target}(\theta ,N,q)}\left[{\mathbf{1}}_{y=y^{\ast }(q)}\right]\right),\end{array}\end{array}$$

where $y^{\ast }(q)$ denotes the ground-truth correct response for $q$, and ${\theta }_{q,y^{\ast }(q)}^{\ast }(N)$ represents the test-time compute-optimal scaling strategy for the problem $q$ with compute budget $N$.

其中，$y^{\ast }(q)$ 表示问题 $q$ 的真实正确回答，${\theta }_{q,y^{\ast }(q)}^{\ast }(N)$ 表示在计算预算为 $N$ 时，问题 $q$ 的测试时计算最优扩展策略。

3.2. Estimating Question Difficulty for Compute-Optimal Scaling

In order to effectively analyze the test-time scaling properties of the different mechanisms discussed in Section 2 (e.g. the proposal distribution and the verifier), we will prescribe an approximation to this optimal strategy ${\theta }_{q,y^{\ast }(q)}^{\ast }(N)$ as a function of a statistic of a given prompt. This statistic estimates a notion of difficulty for a given prompt. The compute-optimal strategy is defined as a function of the difficulty of this prompt. Despite being only an approximate solution to the problem shown in Equation 1, we find that it can still induce substantial improvements in performance over a baseline strategy of allocating this inference-time compute in an ad-hoc or uniformly-sampled manner.

为了有效分析 第 2 节 中讨论的不同机制（例如提议分布和验证器）的测试时扩展性质，我们将为最优策略 ${\theta }_{q,y^{\ast }(q)}^{\ast }(N)$ 给出一种近似，并将其表示为给定提示的某个统计量的函数。 这一统计量估计给定提示的 难度。 计算最优策略被定义为该提示难度的函数。 尽管它只是 公式 1 所示问题的近似解，我们发现，相比临时指定或均匀采样式地分配推理时计算的基线策略，它仍能显著改善性能。

Our estimate of the question difficulty assigns a given question to one of five difficulty levels. We can then use this discrete difficulty categorization to estimate ${\theta }_{q,y^{\ast }(q)}^{\ast }(N)$ on a validation set for a given test-time compute budget. We then apply these compute-optimal strategies on the test-set. Concretely, we select the best performing test-time compute strategy for each difficulty bin independently. In this way, question difficulty acts as a sufficient statistic of a question when designing the compute-optimal strategy.

我们的问题难度估计将给定问题归入五个难度等级之一。 随后，对于给定的测试时计算预算，我们可以利用这一离散难度分类，在验证集上估计 ${\theta }_{q,y^{\ast }(q)}^{\ast }(N)$。 之后，我们将这些计算最优策略应用于测试集。 具体而言，我们为每个难度组独立选择表现最好的测试时计算策略。 这样，在设计计算最优策略时，问题难度就充当了该问题的充分统计量。

Defining difficulty of a problem. Following the approach of Lightman et al., we define question difficulty as a function of a given base LLM. Specifically, we bin the model's pass@1 rate – estimated from 2048 samples – on each question in the test set into five quantiles, each corresponding to increasing difficulty levels. We found this notion of model-specific difficulty bins to be more predictive of the efficacy of using test-time compute in contrast to the hand-labeled difficulty bins in the MATH dataset.

定义问题难度。 遵循 Lightman 等人的方法，我们将问题难度定义为给定基础 LLM 的函数。 具体而言，我们用 2048 个样本估计模型在测试集每个问题上的 pass@1，并将这些通过率划分为五个分位数组，分别对应递增的难度等级。 我们发现，与 MATH 数据集中人工标注的难度分组相比，这种针对具体模型定义的难度分组，更能预测测试时计算的有效性。

That said, we do note that assessing a question's difficulty as described above assumes oracle access to a ground-truth correctness checking function, which is of course not available upon deployment where we are only given access to test prompts that we don't know the answer to. In order to be feasible in practice, a compute-optimal scaling strategy conditioned on difficulty needs to first assess difficulty and then utilize the right scaling strategy to solve this problem. Therefore, we approximate the problem's difficulty via a model-predicted notion of difficulty, which performs the same binning procedure over the the averaged final answer score from a learned verifier (and not groundtruth answer correctness checks) on the same set of 2048 samples per problem. We refer to this setting as model-predicted difficulty and the setting which relies on the ground-truth correctness as oracle difficulty.

不过，我们注意到，上述问题难度评估假定可以像预言机一样访问真实正确性检查函数；而在实际部署中，我们只能接触答案未知的测试提示，自然无法获得这种能力。 为了在实践中可行，以难度为条件的计算最优扩展策略必须先评估难度，再使用合适的扩展策略解决问题。 因此，我们使用 模型预测的难度 来近似问题难度：仍然针对每个问题的同一组 2048 个样本，但依据学习得到的验证器给出的最终答案平均分数，而非真实答案正确性检查，进行相同的分组过程。 我们将这一设置称为 模型预测难度，将依赖真实正确性信息的设置称为 预言机难度。

While model-predicted difficulty removes the need for need knowing the ground truth label, estimating difficulty in this way still incurs additional computation cost during inference. That said, this one-time inference cost can be subsumed within the cost for actually running an inference-time strategy (e.g., when using a verifier, one could use the same inference computation for also running search). More generally, this is akin to exploration-exploitation tradeoff in reinforcement learning: in actual deployment conditions, we must balance the compute spent in assessing difficulty vs applying the most compute-optimal approach. This is a crucial avenue for future work (see Section 8) and our experiments do not account for this cost largely for simplicity, since our goal is to present some of the first results of what is in fact possible by effectively allocating test-time compute.

尽管模型预测难度消除了获知真实标签的需求，但以这种方式估计难度仍会在推理期间产生额外计算成本。 不过，这一一次性的推理成本可以包含在实际运行推理时策略的成本中；例如，在使用验证器时，同一批推理计算也可以用于执行搜索。 更一般地说，这类似于强化学习中的探索与利用权衡：在实际部署条件下，必须平衡用于评估难度的计算，以及用于执行计算最优方法的计算。 这是未来研究的重要方向（见 第 8 节）；我们的实验主要出于简化考虑，没有计入这一成本，因为我们的目标是给出一些初步结果，展示有效分配测试时计算 究竟能够做到什么。

So as to avoid confounders with using the same test set for computing difficulty bins and for selecting the compute-optimal strategy, we use two-fold cross validation on each difficulty bin in the test set. We select the best-performing strategy according to performance on one fold and then measure performance using that strategy on the other fold and vice versa, averaging the results of the two test folds.

为了避免在同一测试集上既计算难度分组又选择计算最优策略所带来的混杂因素，我们在测试集的每个难度组内使用两折交叉验证。 我们根据其中一折的表现选择最佳策略，再在另一折上测量该策略的表现；随后交换两折的角色，并对两次测试结果取平均。

4. Experimental Setup

We first outline our experimental setup for conducting this analysis with multiple verifier design choices and proposal distributions, followed by the analysis results in the subsequent sections.

我们首先概述在多种验证器设计选择和提议分布下开展分析的实验设置，随后在后续章节中给出分析结果。

Datasets. We expect test-time compute to be most helpful when models already have all the basic “knowledge” needed to answer a question, and instead the primary challenge is about drawing (complex) inferences from this knowledge. To this end, we focus on the MATH benchmark, which consists of high-school competition level math problems with a range of difficulty levels. For all experiments, we use the dataset split consisting of 12k train and 500 test questions, used in Lightman et al.

数据集。 我们预计，当模型已经具备回答问题所需的全部基础“知识”，而主要挑战在于根据这些知识进行复杂推断时，测试时计算最有帮助。 为此，我们聚焦 MATH 基准，该基准由不同难度等级的高中竞赛水平数学题组成。 所有实验均采用 Lightman 等人使用的数据划分，包含 12k 道训练题和 500 道测试题。

Models. We conduct our analysis using the PaLM 2-S* (Codey) base model. We believe this model is representative of the capabilities of many contemporary LLMs, and therefore think that our findings likely transfer to similar models. Most importantly, this model attains a non-trivial performance on MATH and yet has not saturated, so we expect this model to provide a good test-bed for us.

模型。 我们使用 PaLM 2-S*（Codey）基础模型开展分析。 我们认为，该模型能够代表许多同时期 LLM 的能力，因此我们的发现很可能可以迁移到类似模型。 最重要的是，该模型在 MATH 上已经取得一定表现，但尚未达到饱和，因此我们预计它能够提供一个合适的实验平台。

5. Scaling Test-Time Compute via Verifiers

In this section we analyze how test-time compute can be scaled by optimizing a verifier, as effectively as possible. To this end, we study different approaches for performing test-time search with process verifiers (PRMs) and analyze the test-time compute scaling properties of these different approaches.

本节分析如何通过优化验证器，尽可能有效地扩展测试时计算。 为此，我们研究使用过程验证器（PRM）进行测试时搜索的不同方法，并分析这些方法的测试时计算扩展性质。

5.1. Training Verifiers Amenable to Search

PRM training. Originally PRM training used human crowd-worker labels. While Lightman et al. released their PRM training data (i.e., the PRM800k dataset), we found this data to be largely ineffective for us. We found that it was easy to exploit a PRM trained on this dataset via even naïve strategies such as best-of-N sampling. We hypothesize that this is likely a result of the distribution shift between the GPT-4 generated samples in their dataset and our PaLM 2 models. Rather than proceeding with the expensive process of collecting crowd-worker PRM labels for our PaLM 2 models, we instead apply the approach of Wang et al. to supervise PRMs without human labels, using estimates of per-step correctness obtained from running Monte Carlo rollouts from each step in the solution.

PRM 训练。 最初，PRM 训练使用人工众包标注。 尽管 Lightman 等人发布了 PRM 训练数据（即 PRM800k 数据集），但我们发现这些数据对我们的设置基本无效。 我们发现，即使采用 best-of-N 采样这样朴素的策略，也很容易利用在该数据集上训练的 PRM 的漏洞。 我们推测，这很可能源于其数据集中 GPT-4 生成的样本与我们的 PaLM 2 模型之间的分布偏移。 我们没有为 PaLM 2 模型开展代价高昂的 PRM 众包标注收集，而是采用 Wang 等人的方法：从解答的每个步骤出发进行蒙特卡洛展开，估计各步骤的正确性，从而在没有人工标签的情况下监督 PRM。

Our PRM's per-step predictions therefore correspond to value estimates of reward-to-go for the base model's sampling policy, similar to recent work. We also compared to an ORM baseline (Appendix F) but found that our PRM consistently outperforms the ORM. Hence, all of the search experiments in this section use a PRM model. Additional details on PRM training are shown in Appendix D.

因此，与近期工作类似，我们的 PRM 逐步预测对应于基础模型采样策略的后续累计奖励价值估计。 我们也与 ORM 基线进行了比较（见 附录 F），但发现我们的 PRM 始终优于 ORM。 因此，本节所有搜索实验均使用 PRM 模型。 有关 PRM 训练的更多细节见 附录 D。

Answer aggregation. At test time, process-based verifiers can be used to score each individual step in a set of solutions sampled from the base model. In order to select the best-of-N answers with the PRM, we need a function that can aggregate across all the per-step scores for each answer to determine the best candidate for the correct answer. To do this, we first aggregate each individual answer's per-step scores to obtain a final score for the full answer (step-wise aggregation). We then aggregate across answers to determine the best answer (inter-answer aggregation). Concretely, we handle step-wise and inter-answer aggregation as follows:

答案聚合。 在测试时，基于过程的验证器可以对从基础模型采样的一组解答中的各个步骤进行评分。 为了使用 PRM 从 N 个答案中选择最佳答案，我们需要一个函数，将每个答案的所有逐步评分聚合起来，从而确定最可能正确的候选答案。 为此，我们先聚合每个答案内部的逐步评分，得到完整答案的最终分数，即步骤级聚合。 随后，我们在不同答案之间进行聚合，确定最佳答案，即答案间聚合。 具体而言，步骤级聚合与答案间聚合的处理方式如下：

• Step-wise aggregation. Rather than aggregating the per-step scores by taking the product or minimum, we instead use the PRM's prediction at the last step as the full-answer score. We found this to perform the best out of all aggregation methods we studied (see Appendix E).
• Inter-answer aggregation. We follow Li et al. and apply “best-of-N weighted” selection rather than standard best-of-N. Best-of-N weighted selection marginalizes the verifier's correctness scores across all solutions with the same final answer, selecting final answer with the greatest total sum.

• 步骤级聚合。 我们没有通过乘积或最小值聚合逐步评分，而是使用 PRM 在最后一步的预测作为完整答案的分数。我们发现，在所研究的所有聚合方法中，这一方法表现最好（见 附录 E）。
• 答案间聚合。 我们遵循 Li 等人的方法，使用“加权 best-of-N”选择，而非标准 best-of-N。加权 best-of-N 将所有最终答案相同的解答的验证器正确性分数相加，进行边缘化，并选择总分最高的最终答案。

5.2. Search Methods Against a PRM

图2：比较不同的 PRM 搜索方法。左：best-of-N 采样 N 个完整答案，再根据 PRM 的最终评分选择最优答案。中：束搜索在每个步骤采样 N 个候选，再根据 PRM 选择前 M 个候选继续搜索。右：前瞻搜索扩展了束搜索的每个步骤，在评估应保留哪些步骤并继续搜索时使用 k 步前瞻，因此需要更多计算。

We optimize the PRM at test time via search methods. We study three search approaches that sample outputs from a few-shot prompted base LLM (see Appendix G). An illustration is shown in Figure 2.

我们在测试时通过搜索方法优化 PRM。 我们研究三种搜索方法，它们都从经过少样本提示的基础 LLM 中采样输出（见 附录 G）。 示意见 图2。

Best-of-N weighted. We sample N answers independently from the base LLM and then select the best answer according to the PRM's final answer judgement.

加权 best-of-N。 我们从基础 LLM 中独立采样 N 个答案，再根据 PRM 对最终答案的判断，选出最佳答案。

Beam search. Beam search optimizes the PRM by searching over its per-step predictions. Our implementation is similar to BFS-V. Concretely, we consider a fixed number of beams $N$ and a beam width $M$. We then run the following steps:

束搜索。 束搜索通过搜索 PRM 的逐步预测来优化 PRM。 我们的实现类似于 BFS-V。 具体而言，我们采用固定的束数量 $N$ 和束宽 $M$。 随后执行以下步骤：

1. sample $N$ initial predictions for the first step in the solution
2. score the generated steps according to the PRM's predicted step-wise reward-to-go estimate (which also corresponds to the total reward from the prefix since the reward is sparse in this setting)
3. filter for only the top $\frac{N}{M}$ highest scoring steps
4. now from each candidate, sample $M$ proposals from the next step, resulting in a total of $N/M\times M$ candidate prefixes again. Then repeat steps 2-4 again.

1. 为解答的第一步采样 $N$ 个初始预测。
2. 根据 PRM 对逐步后续累计奖励的估计，为生成的步骤评分；由于该设置中的奖励是稀疏的，这也对应于从当前前缀出发的总奖励。
3. 仅保留分数最高的前 $\frac{N}{M}$ 个步骤。
4. 现在，从每个候选出发，为下一步采样 $M$ 个提议，再次得到总计 $N/M\times M$ 个候选前缀。然后重复步骤 2–4。

We run this algorithm until the end of a solution or the maximum number of rounds of beam expansion are attained (40 in our case). We conclude the search with N final answer candidates, to which we apply best-of-N weighted selection described above to make our final answer prediction.

我们持续运行该算法，直到解答结束，或达到束扩展轮数的上限；在我们的设置中，该上限为 40。 搜索结束时得到 N 个最终答案候选，再应用前述加权 best-of-N 选择，产生最终答案预测。

Lookahead search. Lookahead search modifies how beam search evaluates individual steps. It uses lookahead rollouts to improve the accuracy of the PRM's value estimation in each step of the search process. Specifically, at each step in the beam search, rather than using the PRM score at the current step to select the top candidates, lookahead search performs a simulation, rolling out up to $k$ steps further while stopping early if the end of solution is reached. To minimize variance in the simulation rollout, we perform rollouts using temperature 0. The PRM's prediction at the end of this rollout is then used to score the current step in the beam search.

前瞻搜索。 前瞻搜索修改了束搜索评估单个步骤的方式。 它通过前瞻展开，提高搜索过程中每一步的 PRM 价值估计准确性。 具体而言，在束搜索的每一步，前瞻搜索不会直接使用当前步骤的 PRM 分数选择最优候选，而是执行模拟，最多向后展开 $k$ 步；若提前到达解答末尾，则提前停止。 为了尽量减小模拟展开的方差，我们采用温度 0 进行展开。 随后，使用 PRM 在该展开末尾的预测，为束搜索的当前步骤评分。

That is, in other words, we can view beam search as a special case of lookahead search with $k=0$. Given an accurate PRM, increasing $k$ should improve the accuracy of the per-step value estimates at the cost of additional compute. Also note that this version of lookahead search is a special case of MCTS, wherein the stochastic elements of MCTS, designed to facilitate exploration, are removed since the PRM is already trained and is frozen. These stochastic elements are largely useful for learning the value function (which we've already learned with our PRM), but less useful at test-time when we want to exploit rather than explore. Therefore, lookahead search is largely representative of how MCTS-style methods would be applied at test-time.

换句话说，可以将束搜索视为 $k=0$ 时前瞻搜索的特例。 在 PRM 准确的前提下，增加 $k$ 应当能够以额外计算为代价，提高逐步价值估计的准确性。 还需注意，这个版本的前瞻搜索也是蒙特卡洛树搜索（MCTS）的一个特例；由于 PRM 已经训练完成且保持冻结，我们移除了 MCTS 中用于促进探索的随机性因素。 这些随机性因素主要有助于学习价值函数，而我们的 PRM 已经完成了这种学习；在测试时，我们更希望利用已有知识而非探索，因此这些因素的作用较小。 所以，前瞻搜索基本可以代表 MCTS 类方法在测试时的使用方式。

图3：左：比较依据 PRM 验证器进行搜索的不同方法。在较低的生成预算下，束搜索表现最好；但随着预算进一步增大，其改进逐渐减弱，最终低于 best-of-N 基线。在相同生成预算下，前瞻搜索通常不如其他方法。右：按难度分组比较束搜索和 best-of-N。每个难度组中的四根柱对应递增的测试时计算预算（4、16、64 和 256 次生成）。在较简单的问题（第 1、2 组）上，束搜索在较高预算下表现出过度优化的迹象，而 best-of-N 没有。在中等难度问题（第 3、4 组）上，束搜索持续优于 best-of-N。

5.3. Analysis Results: Test-Time Scaling for Search with Verifiers

We now present our results comparing various search algorithms and identify a prompt difficulty dependent compute-optimal scaling strategy for search methods.

下面给出不同搜索算法的比较结果，并确定一种依赖提示难度的、适用于搜索方法的计算最优扩展策略。

Comparing search algorithms. We first conduct a sweep over various search settings. In addition to the standard best-of-N approach, we sweep over the two main parameters that distinguish different tree-search methods: beam-width $M$ and number of lookahead steps $k$. While we are not able to extensively sweep every single configuration, we sweep over the following settings with a maximum budget of 256:

比较搜索算法。 我们首先扫描多种搜索设置。 除标准 best-of-N 方法外，我们扫描了区分不同树搜索方法的两个主要参数：束宽 $M$ 和前瞻步数 $k$。 虽然无法对每个配置都进行全面扫描，但我们在最大预算为 256 的条件下考察了以下设置：

1. Beam search with the beam width set to $\sqrt{N}$, where $N$ is the generation budget.
2. Beam search with a fixed beam width of 4.
3. Lookahead search with $k=3$ applied to both beam-search settings 1) and 2).
4. Lookahead search with $k=1$ applied to beam-search setting 1).

1. 束宽设为 $\sqrt{N}$ 的束搜索，其中 $N$ 为生成预算。
2. 束宽固定为 4 的束搜索。
3. 对束搜索设置 1）和 2）均应用 $k=3$ 的前瞻搜索。
4. 对束搜索设置 1）应用 $k=1$ 的前瞻搜索。

To compare search methods as a function of generation budget fairly, we build a protocol for estimating the cost of each method. We consider a generation to be a sampled answer from the base LLM. For beam search and best-of-N the generation budget corresponds to the number of beams and $N$ respectively. Lookahead search, however, utilizes additional compute: at each step of the search, we sample $k$ additional steps ahead. Therefore, we define the cost of lookahead-search to be $N\times (k+1)$ samples.

为了在不同生成预算下公平比较搜索方法，我们建立了一套估计各方法成本的方案。 我们将从基础 LLM 中采样一个答案视为一次生成。 对于束搜索和 best-of-N，生成预算分别对应于束的数量和 $N$。 但前瞻搜索需要额外计算：在搜索的每一步，我们都额外向前采样 $k$ 步。 因此，我们将前瞻搜索的成本定义为 $N\times (k+1)$ 次采样。

Results. As shown in Figure 3 (left), with smaller generation budgets, beam search significantly outperforms best-of-N. However, as the budget is scaled up, these improvements greatly diminish, with beam search often underperforming the best-of-N baseline. We also see that, lookahead-search generally underperforms other methods at the same generation budget, likely due to the additional computation inducted by simulating the lookahead rollouts. The diminishing returns from search are likely due to exploitation of the PRM's predictions.

结果。 如 图3 左侧所示，在较小的生成预算下，束搜索显著优于 best-of-N。 然而，随着预算增大，这些改进显著减弱，束搜索经常不如 best-of-N 基线。 我们还发现，在相同生成预算下，前瞻搜索通常不如其他方法，这很可能是因为模拟前瞻展开引入了额外计算。 搜索收益递减，很可能是由于搜索利用了 PRM 预测中的漏洞。

For example, we see some instances (such as in Figure 29), where search causes the model to generate low-information repetitive steps at the end of a solution. In other cases, we find that over-optimizing search can result in overly short solutions consisting of just 1-2 steps. This explains why the most powerful search method (i.e., lookahead search) underperforms the most. We include several of these examples found by search in Appendix M.

例如，我们观察到一些实例（如 图29），搜索导致模型在解答末尾生成信息量很低的重复步骤。 在另一些情况下，我们发现搜索中的过度优化会导致解答过短，仅包含 1–2 个步骤。 这解释了为什么最强的搜索方法，即前瞻搜索，反而表现最差。 我们在 附录 M 中提供了若干搜索得到的此类例子。

Which problems does search improve? To understand how to compute-optimally scale search methods, we now conduct a difficulty bin analysis. Specifically, we compare beam-search ($M=4$) against best-of-N. In Figure 3 (right) we see that while in aggregate, beam search and best-of-N perform similarly with a high generation budget, evaluating their efficacy over difficulty bins reveals very different trends. On the easy questions (levels 1 and 2), the stronger optimizer of the two approaches, beam search, degrades performance as the generation budget increases, suggesting signs of exploitation of the PRM signal. In contrast, on the harder questions (levels 3 and 4), beam search consistently outperforms best-of-N. On the most difficult questions (level 5), no method makes much meaningful progress.

搜索改善了哪些问题？ 为了理解如何以计算最优的方式扩展搜索方法，我们进行按难度分组的分析。 具体而言，我们比较束搜索（$M=4$）与 best-of-N。 在 图3 右侧可以看到，虽然在较高生成预算下，束搜索与 best-of-N 的总体表现相近，但按难度分组评估其效果后，会呈现非常不同的趋势。 在简单问题（第 1、2 级）上，两种方法中优化力度更强的束搜索，会随着生成预算增加而表现下降，这显示出利用 PRM 信号漏洞的迹象。 相比之下，在较难问题（第 3、4 级）上，束搜索始终优于 best-of-N。 在最困难的问题（第 5 级）上，没有任何方法取得多少有意义的进展。

These findings match intuition: we might expect that on the easy questions, the verifier will make mostly correct assessments of correctness. Therefore, by applying further optimization via beam search, we only further amplify any spurious features learned by the verifier, causing performance degredation. On the more difficult questions, the base model is much less likely to sample the correct answer in the first place, so search can serve to help guide the model towards producing the correct answer more often.

这些发现符合直觉：对于简单问题，我们预期验证器对正确性的判断大多是准确的。 因此，继续通过束搜索加强优化，只会进一步放大验证器学到的虚假特征，导致性能下降。 对于更困难的问题，基础模型最初就采样到正确答案的可能性低得多，因此搜索能够起到引导作用，让模型更频繁地产生正确答案。

图4：比较 PRM 搜索中的计算最优测试时计算分配与基线。依据问题难度扩展测试时计算，可以在仅使用至多 1/4 的测试时计算时，几乎能超过 PRM best-of-N（例如 16 次与 64 次生成）。“Compute-optimal oracle”使用由真实正确性信息得到的预言机难度分组；“compute-optimal predicted”使用 PRM 预测生成难度分组。两种难度分组对应的曲线大体重合。

Compute-optimal search. Given the above results, it is clear that question difficulty can be a useful statistic to predict the optimal search strategy to use at a given compute budget. Additionally, the best choice of search strategy can vary drastically as a function of this difficulty statistic. We therefore visualize the “compute-optimal” scaling trend, as represented by the best performing search strategy at each difficulty level in Figure 4. We see that in the low generation budget regime, using both the oracle and predicted difficulty, compute-optimal scaling can nearly outperform best-of-N using up to 4x less test-time compute (e.g. 16 verses 64 generations). While in the higher budget regime, some of these benefits diminish with the use of predicted difficulty, with oracle bins we still see continued improvements from optimally scaling test-time compute. This result demonstrates the performance gains that could be obtained by adaptively allocating test-time compute during search.

计算最优搜索。 根据以上结果，显然，问题难度可以作为一个有用的统计量，用来预测在给定计算预算下应采用的最优搜索策略。 此外，最佳搜索策略可能随着这一难度统计量发生很大变化。 因此，我们在 图4 中展示“计算最优”的扩展趋势，即每个难度等级上表现最好的搜索策略。 可以看到，在较低生成预算下，无论使用预言机难度还是预测难度，计算最优扩展都能在仅使用至多 1/4 的测试时计算时，几乎能超过 best-of-N（例如 16 次与 64 次生成）。 在较高预算下，使用预测难度时部分收益有所减弱；但采用预言机分组时，最优扩展测试时计算仍然带来持续改进。 这一结果展示了在搜索过程中自适应分配测试时计算所能带来的性能收益。

Takeaways for compute-optimal scaling of verifiers

We find that the efficacy of any given verifier search method depends critically on both the compute budget and the question at hand. Specifically, beam-search is more effective on harder questions and at lower compute budgets, whereas best-of-N is more effective on easier questions and at higher budgets. Moreover, by selecting the best search setting for a given question difficulty and test-time compute budget, we can nearly outperform best-of-N using up to 4x less test-time compute.

验证器计算最优扩展的要点

我们发现，任何给定验证器搜索方法的效果，都高度依赖于计算预算和当前问题。 具体而言，束搜索在较难问题和较低计算预算下更有效，而 best-of-N 在较简单问题和较高预算下更有效。 此外，针对给定问题难度和测试时计算预算选择最佳搜索设置，可以在仅使用至多 1/4 的测试时计算时，几乎能超过 best-of-N。

6. Refining the Proposal Distribution

So far, we studied the test-time compute scaling properties of search against verifiers. Now we turn to studying the scaling properties of modifying the proposal distribution (Section 2). Concretely, we enable the model to revise their own answers iteratively, allowing the model to dynamically improve it's own distribution at test time. Simply prompting existing LLMs to correct their own mistakes tends to be largely ineffective for obtaining performance improvements on reasoning problems.

到目前为止，我们研究了依据验证器进行搜索的测试时计算扩展性质。 现在，我们转而研究修改提议分布的扩展性质（见 第 2 节）。 具体而言，我们让模型能够迭代修订自身答案，从而在测试时动态改进自身分布。 仅仅提示现有 LLM 纠正自身错误，通常很难有效改善推理问题上的表现。

Therefore, we build on the recipe prescribed by Qu et al., incorporate modifications for our setting, and finetune language models to iteratively revise their own answers. We first describe how we train and use models that refine their own proposal distribution by sequentially conditioning on their own previous attempts at the question. We then analyze the inference-time scaling properties of revision models.

因此，我们以 Qu 等人提出的方法为基础，根据我们的设置作出调整，并微调语言模型，使其能够迭代修订自身答案。 我们首先介绍如何训练和使用这种模型：让模型依次以自己此前对同一问题的尝试为条件，改进自身提议分布。 随后，我们分析修订模型的推理时扩展性质。

图5：并行采样（例如 best-of-N）与顺序修订。左：并行采样独立、并行地生成 N 个答案；顺序修订则以先前尝试为条件，依次生成每个答案。右：在顺序与并行两种情况下，都可以使用验证器从 N 个答案中选出最佳答案（例如采用加权 best-of-N）。还可以将部分预算用于并行采样，另一部分用于顺序修订，从而组合两种采样策略。在这种情况下，先用验证器选出每条顺序链中的最佳答案，再在各条链之间选出最佳答案。

6.1. Setup: Training and Using Revision Models

Our procedure for finetuning revision models is similar to Qu et al., though we introduce some crucial differences. For finetuning, we need trajectories consisting of a sequence of incorrect answers followed by a correct answer, that we can then run SFT on. Ideally, we want the correct answer to be correlated with the incorrect answers provided in context, so as to effectively teach the model to implicitly identify mistakes in examples provided in-context, followed by correcting those mistakes by making edits as opposed to ignoring the in-context examples altogether, and trying again from scratch.

我们的修订模型微调流程与 Qu 等人的方法相似，但引入了若干关键差异。 为了进行微调，我们需要由一连串错误答案和其后一个正确答案组成的轨迹，然后在这些轨迹上进行监督微调（SFT）。 理想情况下，我们希望正确答案与上下文中给出的错误答案具有 相关性，以便有效地教会模型 隐式地 识别上下文示例中的错误，再通过编辑纠正这些错误，而不是完全忽略上下文中的示例、从头再试。

Generating revision data. The on-policy approach of Qu et al. for obtaining several multi-turn rollouts was shown to be effective, but it was not entirely feasible in our infrastructure due to compute costs associated with running multi-turn rollouts. Therefore, we sampled 64 responses in parallel at a higher temperature and post-hoc constructed multi-turn rollouts from these independent samples. Specifically, following the recipe of Training revision models with synthetic data (2024), we pair up each correct answer with a sequence of incorrect answers from this set as context to construct multi-turn finetuning data. We include up to four incorrect answers in context, where the specific number of solutions in context is sampled randomly from a uniform distribution over categories 0 to 4. We use a character edit distance metric to prioritize selecting incorrect answers which are correlated with the final correct answer (see Appendix H). Note that token edit distance is not a perfect measure of correlation, but we found this heuristic to be sufficient to correlate incorrect in-context answers with correct target answers to facilitate training a meaningful revision model, as opposed to randomly pairing incorrect and correct responses with uncorrelated responses.

生成修订数据。 Qu 等人获取多条多轮展开轨迹的同策略方法已被证明有效，但运行多轮展开所需的计算成本，使其在我们的基础设施上并不完全可行。 因此，我们采用较高温度 并行 采样 64 个回答，再事后从这些独立样本构造多轮展开轨迹。 具体而言，遵循《Training revision models with synthetic data》（2024）的方法，我们将每个正确答案与该集合中的一系列错误答案配对，并将后者作为上下文，构造多轮微调数据。 上下文中最多包含四个错误答案，具体包含多少个解答，从 0 到 4 的类别均匀分布中随机采样。 我们使用字符编辑距离指标，优先选择与最终正确答案相关的错误答案（见 附录 H）。 注意，token 编辑距离并不是相关性的完美度量，但我们发现，这一启发式方法已足以将上下文中的错误答案与正确目标答案关联起来，从而训练出有意义的修订模型，而非随机将互不相关的错误回答和正确回答配对。

Using revisions at inference-time. Given a finetuned revision model, we can then sample a sequence of revisions from the model at test-time. While our revision model is only trained with up to four previous answers in-context, we can sample longer chains by truncating the context to the most recent four revised responses. In Figure 6 (left), we see that as we sample longer chains from the revision model, the model's pass@1 at each step gradually improves, demonstrating that we are able to effectively teach the model to learn from mistakes made by previous answers in context.

在推理时使用修订。 得到微调后的修订模型后，我们可以在测试时从模型中采样一系列修订。 尽管训练时模型的上下文最多只包含四个先前答案，但通过将上下文截断为最近四个修订后的回答，我们能够采样更长的链。 在 图6 左侧可以看到，随着从修订模型采样的链变长，模型在每一步的 pass@1 逐渐提高，说明我们能够有效地教会模型从上下文中先前答案的错误里学习。

图6：左：修订模型在每个修订步骤的 pass@1。随着修订步骤增加，pass@1 逐渐提高，甚至在超过训练时的 4 次修订后仍继续提高。对于测试集中的每个问题，我们对 4 条长度为 64 的修订轨迹的表现取平均，以估计各步骤的 pass@1。右：修订模型的顺序采样与并行采样。比较从修订模型并行生成 N 个初始答案，与顺序生成 N 次修订的表现。无论采用验证器还是多数投票来选择答案，修订模型顺序生成答案的效果都略优于并行生成。

That said, there is a distribution shift at inference time: the model was trained on only sequences with incorrect answers in context, but at test-time the model may sample correct answers that are included in the context. In this case, it may incidentally turn the correct answer into an incorrect answer in the next revision step. We find that indeed, similar to Qu et al., around 38% of correct answers get converted back to incorrect ones with our revision model using a naïve approach. Therefore, we employ a mechanism based on sequential majority voting or verifier-based selection to select the most correct answer from the sequence of revisions made by the model (see Figure 5) to produce the best answer.

不过，推理时存在分布偏移：训练序列的上下文只包含错误答案，但在测试时，模型可能采样到正确答案，并将其纳入上下文。 在这种情况下，它可能在下一次修订时无意中把正确答案改成错误答案。 我们确实发现，与 Qu 等人的观察类似，对修订模型采用朴素方法时，约 38% 的正确答案会被重新改错。 因此，我们采用基于顺序多数投票或验证器选择的机制，从模型的一系列修订中选出最可能正确的答案（见 图5），作为最终的最佳答案。

Comparisons. To test the efficacy of modifying the proposal distribution via revisions, we setup an even comparison between the performance of sampling N revisions in sequence and sampling N attempts at a question in parallel. We see in Figure 6 (right), that with both the verifier-based and majority-based selection mechanisms sampling solutions in sequence outperforms sampling them in parallel.

比较。 为了检验通过修订改变提议分布的效果，我们公平比较了依次采样 N 次修订与并行采样 N 次解题尝试的表现。 从 图6 右侧可以看到，无论使用基于验证器还是基于多数投票的选择机制，顺序采样解答都优于并行采样。

图7：左：改变生成预算分配给顺序修订与并行采样的比例。每条曲线对应一个固定生成预算，横轴改变两者比例，使用验证器选择答案。尽管增加顺序修订通常优于增加并行计算，但在较高生成预算下，存在一个在两种极端之间取得平衡的理想比例。右：在不同难度分组中，对 128 的生成预算改变顺序与并行之比。使用验证器选择答案时，较简单的问题在全部采用顺序计算时表现最佳；较难问题则存在一个理想的顺序与并行测试时计算比例。

6.2. Analysis Results: Test-Time Scaling with Revisions

We saw previously that proposing answers sequentially outperforms proposing them in parallel. However, we might expect sequential and parallel sampling to have different properties. Sampling answers in parallel may act as more of a global search process, that could in principle, provide coverage over many totally different approaches for solving a problem, for instance, different candidates might utilize different high-level approaches altogether. Sequential sampling, on the other hand, may work more as a local refinement process, revising responses that are already somewhat on the right track. Due to these complementary benefits, we should strike a balance between these two extremes by allocating some of our inference-time budget to parallel sampling (e.g. $\sqrt{N}$) and the rest to sequential revisions (e.g. $\sqrt{N}$). We will now show the existence of a compute-optimal ratio between sequential and parallel sampling, and understand their relative pros and cons based on difficulty of a given prompt.

前面已经看到，顺序提出答案优于并行提出答案。 不过，我们预期顺序采样与并行采样可能具有不同性质。 并行采样答案更像是全局搜索过程，原则上可以覆盖许多完全不同的解题方法；例如，不同候选可能采用截然不同的高层思路。 另一方面，顺序采样更像是局部改进过程，修订那些已经大致沿着正确方向的回答。 由于这些收益具有互补性，我们应当在两种极端之间取得平衡，将部分推理时预算分配给并行采样（例如 $\sqrt{N}$），其余分配给顺序修订（例如 $\sqrt{N}$）。 下面，我们将展示顺序与并行采样之间存在一个计算最优的比例，并根据给定提示的难度，分析两者各自的优缺点。

Trading off sequential and parallel test-time compute. To understand how to optimally allocate sequential and parallel compute, we perform a sweep over a number of different ratios. We see, in Figure 7 (left), that indeed, at a given generation budget, there exists an ideal sequential to parallel ratio, that achieves the maximum accuracy. We also see in Figure 7 (right) that the ideal ratio of sequential to parallel varies depending on a given question's difficulty. In particular, easy questions benefit more from sequential revisions, whereas on difficult questions it is optimal to strike a balance between sequential and parallel computation. This finding supports the hypothesis that sequential revisions (i.e., varying the proposal distribution) and parallel sampling (i.e., search with verifiers) are two complementary axes for scaling up test-time compute, which may be more effective on a per-prompt basis. We include examples of our model's generations in Appendix L. Additional results are shown in Appendix B.

权衡顺序与并行测试时计算。 为了理解如何最优地分配顺序与并行计算，我们扫描了多个不同的比例。 在 图7 左侧可以看到，对于给定生成预算，确实 存在一个能够达到最高准确率的理想顺序与并行比例。 我们还在 图7 右侧看到，理想的顺序与并行比例会随给定问题的难度而变化。 具体而言，简单问题从顺序修订中获益更多，而对于困难问题，在顺序与并行计算之间取得平衡才是最优选择。 这一发现支持如下假设：顺序修订（即改变提议分布）与并行采样（即使用验证器搜索）是扩展测试时计算的两个互补维度，根据每个提示作出选择可能更加有效。 我们在 附录 L 中提供了模型生成结果的示例。 更多结果见 附录 B。

图8：比较修订模型的计算最优测试时计算分配与并行计算基线。根据问题难度最优地扩展测试时计算，可以仅用至多 1/4 的测试时计算便超过 best-of-N（例如 64 次与 256 次采样）。“Compute-optimal oracle”使用由真实正确性信息得到的预言机难度分组；“compute optimal predicted”则使用 PRM 预测生成模型预测的难度分组。

Compute-optimal revisions. Given that the efficacy of sequential and parallel sampling depends on question difficulty, we can select the ideal ratio of sequential to parallel compute per difficulty bin. In Figure 8, we plot results using this compute-optimal scaling strategy when employing both our oracle and predicted notions of difficulty. In both cases, we're able to substantially improve test-time compute scaling by improving the proposal distribution via revisions. In particular, we see that at higher generation budgets, parallel sampling seems to plateau, whereas compute-optimal scaling demonstrates continued improvements. For both oracle and predicted difficulty bins, we see that compute-optimal scaling can outperform best-of-N using up to 4x less test-time compute (e.g. 64 samples verses 256). Overall, these results demonstrate the potential for improved test-time compute scaling by adjusting the proposal distribution on a per-prompt basis.

计算最优修订。 由于顺序与并行采样的有效性依赖于问题难度，我们可以为每个难度组选择理想的顺序与并行计算比例。 在 图8 中，我们分别采用预言机难度和预测难度，展示这一计算最优扩展策略的结果。 在两种情况下，通过修订改进提议分布，我们都能显著改善测试时计算的扩展效果。 特别是，我们观察到在较高生成预算下，并行采样似乎趋于平台期，而计算最优扩展仍持续改善。 对于预言机难度分组和预测难度分组，我们都发现，计算最优扩展能够仅用至多 1/4 的测试时计算就超过 best-of-N（例如 64 次与 256 次采样）。 总体而言，这些结果展示了按提示调整提议分布，从而改善测试时计算扩展效果的潜力。

Takeaways for compute-optimal scaling by refining the proposal distribution with revisions

We find that there exists a tradeoff between sequential (e.g. revisions) and parallel (e.g. standard best-of-N) test-time computation, and the ideal ratio of sequential to parallel test-time compute depends critially on both the compute budget and the specific question at hand. Specifically, easier questions benefit from purely sequential test-time compute, whereas harder questions often perform best with some ideal ratio of sequential to parallel compute. Moreover, by optimally selecting the best setting for a given question difficulty and test-time compute budget, we can outperform the parallel best-of-N baseline using up to 4x less test-time compute.

通过修订改进提议分布以实现计算最优扩展的要点

我们发现，顺序测试时计算（例如修订）与并行测试时计算（例如标准 best-of-N）之间存在权衡，而理想的顺序与并行比例高度依赖于计算预算和当前具体问题。 具体而言，较简单的问题适合完全采用顺序测试时计算，而较难问题通常在某个理想的顺序与并行比例下表现最佳。 此外，针对给定问题难度和测试时计算预算，最优地选择设置，可以仅用至多 1/4 的测试时计算就超过并行 best-of-N 基线。

7. Putting it Together: Exchanging Pretraining and Test-Time Compute

So far, we saw that utilizing additional test-time computation can enable us to represent more complex distributions than the one predicted by the base LLM itself, thereby improving performance. We now posit that this increased flexibility of representing distributions means that we can expect additional test-time compute to make up for the lack of a higher-capacity model or training for more FLOPs during pre-training. In this section, we study to what extent this is possible. We pose the following question:

到目前为止，我们已经看到，额外的测试时计算能够表示比基础 LLM 本身预测的分布更复杂的分布，从而提高性能。 我们进一步认为，分布表达灵活性的提升意味着，额外测试时计算有望弥补模型容量不足，或预训练阶段训练 FLOPs 不足的问题。 本节中，我们研究这在多大程度上能够实现。 我们提出以下问题：

Question: Exchanging pretraining and test-time compute

Suppose a model was pre-trained with $X$ FLOPs. Assume that we plan to run $Y$ FLOPs of inference with this model. If we want to improve performance by increasing the total FLOPs budget by a factor of $M$ (i.e., $M(X+Y)$ total FLOPs across both pretraining and inference), should we spend our FLOPs on increased pretraining compute or on additional test-time compute?

问题：预训练与测试时计算的替换

假设一个模型在预训练时使用了 $X$ FLOPs。 再假设我们计划用该模型执行 $Y$ FLOPs 的推理。 如果希望将总 FLOPs 预算增加至原来的 $M$ 倍来提高性能，即预训练与推理合计使用 $M(X+Y)$ FLOPs，那么应当把这些 FLOPs 用于增加预训练计算，还是增加测试时计算？

Increasing pretraining FLOPS introduces the additional design decision of whether to allocate compute to training with more data or more parameters. We focus on the setting in which model parameters are scaled up and training data amount is fixed, matching the approach taken with the open-source LLaMA series of models. We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work.

增加预训练 FLOPs 会带来一个额外的设计选择：是将计算用于更多数据的训练，还是更多参数的训练。 我们聚焦于扩大模型参数而保持训练数据量固定的设置，这与开源 LLaMA 系列模型采用的方法一致。 我们选择这一设置，是因为它代表了一种典型的预训练计算扩展方式；而数据量与参数量等比例扩展的计算最优预训练扩展分析，则留待未来研究。

Defining an exchange rate between FLOPs. We now describe how we define the exchange rate between pretraining and inference FLOPs. To determine pretraining FLOPs, use use the common approximation $X=6N{D}_{\text{pretrain}}$, and for inference FLOPs, we use $Y=2N{D}_{\text{inference}}$. Here $N$ represents model parameters, $D_{\text{pretrain}}$ is the number of tokens used for pretraining, and $D_{\text{inference}}$ the total number of tokens generated at inference time. With these approximations, we can see that, if we multiply the model parameters by a factor of $M$, then both the pretraining and inference FLOPs (due to the cost of greedy decoding with the larger model), increase by a factor of $M$ (giving $M(X+Y)$ total FLOPs).

定义 FLOPs 之间的兑换率。 下面介绍如何定义预训练 FLOPs 与推理 FLOPs 之间的兑换率。 对于预训练 FLOPs，我们采用常见近似 $X=6N{D}_{\text{pretrain}}$；对于推理 FLOPs，则采用 $Y=2N{D}_{\text{inference}}$。 其中，$N$ 表示模型参数量，$D_{\text{pretrain}}$ 为预训练使用的 token 数，$D_{\text{inference}}$ 为推理时生成的 token 总数。 根据这些近似可以看出，如果将模型参数量乘以 $M$，预训练 FLOPs 与推理 FLOPs 都会乘以 $M$；后者源于较大模型贪心解码的成本，因此总 FLOPs 为 $M(X+Y)$。

图9：FLOPs 匹配评估中预训练与测试时计算之间的权衡。每条曲线表示在一个预言机难度组内采用计算最优策略扩展测试时计算的表现，左侧为修订，右侧为搜索。星号表示一个参数规模约大 14 倍的预训练基础模型采用贪心解码时的 pass@1。横轴为测试时计算预算；星号位于横轴上的三个不同位置，分别对应三种推理计算负载下，扩展参数与扩展测试时计算的 FLOPs 等价比较点，其中 R = 推理 token 数 / 预训练 token 数。若星号低于曲线，意味着使用测试时计算比扩大模型参数更有效；若星号高于曲线，则意味着扩大参数更有效。对于简单问题或推理负载较低的情形（例如 R ≪ 1），测试时计算通常优于扩大模型参数；对于较难问题或推理负载较高的情形（例如 R ≫ 1），预训练是更有效的性能提升方式。

To match the FLOPs from scaling up model paramters using test-time compute with the smaller model we can multiply the smaller model's inference compute by a factor of $M+3\left(\frac{D_{\text{pretrain}}}{D_{\text{inference}}}\right)(M-1)$. Notably, the amount of inference compute we can utilize to match the FLOPs for the larger model depends on the ratio $\frac{D_{\text{pretrain}}}{D_{\text{inference}}}$. We refer to the inverse of this ratio as $R$ (e.g. $\frac{D_{\text{inference}}}{D_{\text{pretrain}}}$). Depending on the specific production setting or use-case, we should expect very different values of $R$.

为了通过较小模型的测试时计算，匹配扩大模型参数所消耗的 FLOPs，可以将较小模型的推理计算量乘以 $M+3\left(\frac{D_{\text{pretrain}}}{D_{\text{inference}}}\right)(M-1)$。 值得注意的是，为匹配较大模型的 FLOPs 所能使用的推理计算量，取决于比值 $\frac{D_{\text{pretrain}}}{D_{\text{inference}}}$。 我们将这一比值的倒数称为 $R$，即 $\frac{D_{\text{inference}}}{D_{\text{pretrain}}}$。 不同的生产设置或应用场景，可能对应非常不同的 $R$ 值。

In particular, in many large scale production settings, we may expect significantly more inference tokens than pretraining tokens, in which case we would have $R>>1$. On the other hand, in many contemporary self-improvement setups, that would use test-time compute to improve the model, we would likely generate significantly fewer inference tokens than pretraining tokens, giving $R<<1$. Therefore, since the scale of test-time compute we can apply is dependent on this ratio, we expect differing conclusions depending on the specific setting.

特别是，在许多大规模生产场景中，预期推理 token 数可能远多于预训练 token 数，此时 $R>>1$。 另一方面，在许多利用测试时计算改进模型的当代自我改进设置中，生成的推理 token 数很可能远少于预训练 token 数，因此 $R<<1$。 由于可以投入的测试时计算规模取决于这一比值，我们预期不同具体设置会得出不同结论。

In Figure 9, we use this approach to exchanging test-time and pretraining compute to compare our compute-optimal scaling against scaling up model parameters by a factor of $\sim 14$. We conduct comparisons for 3 different values of R: 0.16 ($R<<1$), 0.79 ($R\sim 1$), and 22 ($R>>1$), with each ratio corresponding to an inference budget. Observe that if we only expect to see very difficult questions (e.g. difficulty bins 4/5) or have a larger $D_{\text{inference}}$ (corresponding to a larger $R$ value), then it is often more effective to allocate our budget towards pretraining (e.g. the star is above the line). If instead, we expect mostly easy or intermediate difficulty questions (e.g. bins 1/2/3 and sometimes 4) or have lower inference requirements (as is the case in self-improvement pipelines), then utilizing test-time compute is better.

在 图9 中，我们使用这种测试时计算与预训练计算的兑换方式，比较计算最优扩展与将模型参数扩大至 $\sim 14$ 倍的效果。 我们针对 R 的 3 个不同取值进行比较：0.16（$R<<1$）、0.79（$R\sim 1$）和 22（$R>>1$），每个比值对应一个推理预算。 可以看到，如果预期只遇到非常困难的问题（例如难度组 4/5），或 $D_{\text{inference}}$ 较大、对应较大的 $R$，那么将预算分配给预训练往往更有效；此时星号位于曲线上方。 反之，如果预期主要遇到简单或中等难度的问题（例如第 1/2/3 组，有时也包括第 4 组），或推理需求较低（如自我改进流程），那么使用测试时计算更好。

Takeaways for exchanging pretraining and test-time compute

Test-time and pretraining compute are not 1-to-1 “exchangeable”. On easy and medium questions, which are within a model's capabilities, or in settings with small inference requirement, test-time compute can easily cover up for additional pretraining. However, on challenging questions which are outside a given base model's capabilities or under higher inference requirement, pretraining is likely more effective for improving performance.

预训练与测试时计算替换的要点

测试时计算与预训练计算并不能一比一“互换”。 对于处于模型能力范围内的简单和中等难度问题，或推理需求较小的设置，测试时计算很容易弥补额外预训练的不足。 但对于超出给定基础模型能力范围的挑战性问题，或推理需求较高的设置，预训练很可能是更有效的性能提升方式。

8. Discussion and Future Work

In this work, we conducted a thorough analysis of the efficacy of different techniques that aim to either improve search against a verifier or to refine an LLM's proposal distribution, for scaling test-time compute for math reasoning. In general, we found that the efficacy of a given approach heavily correlates with the difficulty of the problem from the perspective of the base LLM's capabilities. This motivated us to introduce the notion of “compute-optimal” scaling of test-time computation, which prescribes a adaptive, prompt-dependent strategy to improve performance under a given test-time compute budget. By applying such a compute-optimal scaling strategy, we find that can improve the efficiency of test-time compute scaling by a factor of $2-4\times$. When comparing benefits obtained from additional test-time compute against benefits from additional pre-training compute in a FLOPs-matched setting, we show for the first time that using test-time computation with seemingly simple methods (i.e., revisions and search) can already scale well on certain types of prompts, providing gains over spending those FLOPs in pretraining. That said, there are also limitations associated with our study that future work can aim to address.

本研究全面分析了两类技术在扩展数学推理测试时计算方面的效果：一类改进依据验证器进行的搜索，另一类改进 LLM 的提议分布。 总体而言，我们发现，从基础 LLM 能力的角度衡量的问题难度，与给定方法的有效性高度相关。 这促使我们引入测试时计算“计算最优”扩展的概念：在给定测试时计算预算下，采用依赖提示的自适应策略来提高性能。 通过应用这一计算最优扩展策略，我们发现可以将测试时计算扩展的效率提升至 2–4 倍。 在 FLOPs 匹配的设置下，将额外测试时计算与额外预训练计算带来的收益进行比较时，我们首次表明，看似简单的测试时计算方法，即修订和搜索，已经能在某些类型的提示上实现良好扩展，效果优于将这些 FLOPs 用于预训练。 不过，本研究也存在一些可由未来工作解决的局限。

Further improving test-time compute scaling. In this work we focused on improving the test-time compute scaling of two primary mechanisms: the verifier and the proposal distribution (via revisions). While we combined verifiers with revisions in Section 6, we did not experiment with PRM tree-search techniques in combination with revisions. Neither did we study other techniques such as critique and revise. Future work should investigate how test-time compute scaling can be further improved by combining a variety of these approaches. Additionally, we found that across the board these schemes provided small gains on hard problems; future work should work to develop new ways of using test-time compute which can circumvent this limitation.

进一步改善测试时计算的扩展。 本研究主要改进两种机制的测试时计算扩展：验证器，以及通过修订改进提议分布。 尽管我们在 第 6 节 中将验证器与修订结合，但并未尝试将 PRM 树搜索技术与修订结合。 我们也没有研究批评后修订等其他技术。 未来工作应研究如何组合这些不同方法，进一步改善测试时计算的扩展。 此外，我们发现这些方案在困难问题上普遍收益较小；未来研究应开发新的测试时计算方式，以突破这一局限。

Assessing question difficulty quickly. We used a notion question difficulty as a simple sufficient statistic for approximating the compute-optimal test-time scaling strategy. While this scheme was effective, estimating our notion of difficulty requires applying a non-trivial amount of test-time compute itself. Future work should consider alternative ways of more efficiently estimating question difficulty (e.g., by pretraining or finetuning models to directly predict difficulty of a question) or dynamically switching between assessing difficulty and attempting to solve a question.

快速评估问题难度。 我们使用问题难度这一概念作为简单的充分统计量，近似计算最优的测试时扩展策略。 虽然这种方案有效，但估计这种难度本身也需要投入相当数量的测试时计算。 未来工作应考虑其他更高效的问题难度估计方法，例如通过预训练或微调，使模型直接预测问题难度；也可以在评估难度与尝试解题之间动态切换。

Interleaving test-time and training-time compute. We focused purely on test-time compute scaling in this work and the degree to which test-time compute can be traded off for additional pretraining. However, in the future, we envision that the outputs of applying additional test-time compute can be distilled back into the base LLM, enabling an iterative self-improvement loop that operates on open-ended natural language. To this end, future work should extend our findings and study how the outputs of applying test-time compute can be used to improve the base LLM itself.

交替进行测试时计算与训练时计算。 本研究仅聚焦于测试时计算扩展，以及测试时计算能够在多大程度上替代额外预训练。 但展望未来，我们设想可以将投入额外测试时计算所产生的输出，重新蒸馏到基础 LLM 中，从而形成一个能够处理开放式自然语言的迭代自我改进循环。 为此，未来工作应扩展我们的发现，研究如何利用测试时计算产生的输出来改进基础 LLM 本身。