s1: Simple test-time scaling
LLMReasoningTest-Time Compute斯坦福大学华盛顿大学Allen Institute for AIContextual AIs1：简单的测试时扩展
Abstract
Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. We seek the simplest approach to achieve test-time scaling and strong reasoning performance. First, we curate a small dataset s1K of 1,000 questions paired with reasoning traces relying on three criteria we validate through ablations: difficulty, diversity, and quality. Second, we develop budget forcing to control test-time compute by forcefully terminating the model's thinking process or lengthening it by appending “Wait” multiple times to the model's generation when it tries to end.
测试时扩展是一种很有前景的新语言建模方法，它利用额外的测试时计算来提升性能。 近期，OpenAI 的 o1 模型展现了这种能力，但并未公开其方法，因此催生了众多复现工作。 我们希望找到最简单的方法，同时实现测试时扩展和强大的推理性能。 首先，我们依据难度、多样性和质量这三个通过消融实验验证的标准，整理出小型数据集 s1K，其中包含 1,000 道问题及其配套的推理轨迹。 其次，我们提出预算强制（budget forcing）来控制测试时计算：强制终止模型的思考过程，或者在模型试图结束生成时多次追加“Wait”，以延长思考。
This can lead the model to double-check its answer, often fixing incorrect reasoning steps. After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). Further, scaling s1-32B with budget forcing allows extrapolating beyond its performance without test-time intervention: from 50% to 57% on AIME24. Our model, data, and code are open-source at https://github.com/simplescaling/s1.
这可以促使模型再次检查答案，往往能够修正错误的推理步骤。 在 s1K 上对 Qwen2.5-32B-Instruct 语言模型进行监督微调，并为其配备预算强制之后，我们的模型 s1-32B 在数学竞赛题（MATH 和 AIME24）上的表现最高超过 o1-preview 27%。 此外，通过预算强制扩展 s1-32B，可以使其超越不进行测试时干预时的性能：在 AIME24 上从 50% 提升到 57%。 我们的模型、数据和代码已开源：https://github.com/simplescaling/s1。
1. Introduction
Performance improvements of language models (LMs) over the past years have largely relied on scaling up train-time compute using large-scale self-supervised pretraining. The creation of these powerful models has set the stage for a new scaling paradigm built on top of them: test-time scaling. The aim of this approach is to increase the compute at test time to get better results. There has been much work exploring this idea, and the viability of this paradigm was recently validated by OpenAI o1. o1 has demonstrated strong reasoning performance with consistent gains from scaling test-time compute.
过去几年，语言模型（LM）的性能提升主要依赖于通过大规模自监督预训练来增加训练时计算。 这些强大模型的出现，为在其基础上建立新的扩展范式——测试时扩展——创造了条件。 这种方法旨在增加测试时的计算量，以获得更好的结果。 已有大量工作探索了这一思路，而 OpenAI o1 最近验证了这一范式的可行性。 o1 展现出强大的推理能力，并随着测试时计算的增加持续获得收益。
OpenAI describes their approach as using large-scale reinforcement learning (RL) implying the use of sizable amounts of data. This has led to various attempts to replicate their models relying on techniques like Monte Carlo Tree Search, multi-agent approaches, and others. Among these approaches, DeepSeek R1 has successfully replicated o1-level performance, also employing reinforcement learning via millions of samples and multiple training stages. However, despite the large number of o1 replication attempts, none have openly replicated a clear test-time scaling behavior. Thus, we ask: what is the simplest approach to achieve both test-time scaling and strong reasoning performance?
OpenAI 将其方法描述为使用大规模强化学习（RL），这意味着需要相当多的数据。 这引发了多种复现尝试，采用的技术包括蒙特卡洛树搜索、多智能体方法，以及其他方法。 其中，DeepSeek R1 成功复现了 o1 级别的性能，同样采用强化学习，并使用数百万个样本和多个训练阶段。 然而，尽管复现 o1 的尝试很多，尚无工作公开复现出清晰的测试时扩展行为。 因此，我们提出一个问题：同时实现测试时扩展和强大推理性能的最简单方法是什么？
We show that training on only 1,000 samples with next-token prediction and controlling thinking duration via a simple test-time technique we refer to as budget forcing leads to a strong reasoning model that scales in performance with more test-time compute. Specifically, we construct s1K, which consists of 1,000 carefully curated questions paired with reasoning traces and answers distilled from Gemini Thinking Experimental. We perform supervised fine-tuning (SFT) of an off-the-shelf pretrained model on our small dataset requiring just 26 minutes of training on 16 H100 GPUs. After training, we control the amount of test-time compute our model spends using budget forcing: (I) If the model generates more thinking tokens than a desired limit, we forcefully end the thinking process by appending an end-of-thinking token delimiter.
我们表明，仅使用 1,000 个样本进行下一 token 预测训练，并通过一种称为 预算强制 的简单测试时技术控制思考时长，就能获得一个推理能力强、且性能随测试时计算增加而提升的模型。 具体而言，我们构建了 s1K，其中包含 1,000 道精心整理的问题，以及从 Gemini Thinking Experimental 蒸馏得到的配套推理轨迹和答案。 我们在这个小型数据集上对一个现成的预训练模型进行监督微调（SFT），在 16 张 H100 GPU 上仅需训练 26 分钟。 训练完成后，我们通过 预算强制 控制模型使用的测试时计算量：(I) 如果模型生成的思考 token 数超过预设上限，我们就追加思考结束 token 分隔符，强制结束思考过程。
Ending the thinking this way makes the model transition to generating its answer. (II) If we want the model to spend more test-time compute on a problem, we suppress the generation of the end-of-thinking token delimiter and instead append “Wait” to the model's current reasoning trace to encourage more exploration. Equipped with this simple recipe – SFT on 1,000 samples and test-time budget forcing – our model s1-32B exhibits test-time scaling (Figure 1). Further, s1-32B is the most sample-efficient reasoning model and outperforms closed-source models like OpenAI's o1-preview (Figure 2).
以这种方式结束思考，会让模型转而生成答案。 (II) 如果希望模型在某个问题上投入更多测试时计算，我们就抑制思考结束 token 分隔符的生成，改为在模型当前的推理轨迹后追加“Wait”，以鼓励进一步探索。 凭借这一简单方案——在 1,000 个样本上进行 SFT，并在测试时施加预算强制——我们的模型 s1-32B 展现出了测试时扩展能力（图1）。 此外，s1-32B 是样本效率最高的推理模型，并且超过了 OpenAI o1-preview 等闭源模型（图2）。
We conduct extensive ablation experiments targeting (a) our selection of 1,000 (1K) reasoning samples and (b) our test-time scaling. For (a), we find that jointly incorporating difficulty, diversity, and quality measures into our selection algorithm is important. Random selection, selecting samples with the longest reasoning traces, or only selecting maximally diverse samples all lead to significantly worse performance (around $-$30% on AIME24 on average). Training on our full data pool of 59K examples, a superset of s1K, does not offer substantial gains over our 1K selection.
我们围绕以下两方面开展了大量消融实验：(a) 对 1,000（1K）个推理样本的选择，以及 (b) 测试时扩展。 对于 (a)，我们发现，在选择算法中同时纳入难度、多样性和质量指标十分重要。 随机选择、选择推理轨迹最长的样本，或仅选择多样性最高的样本，都会导致性能显著下降（在 AIME24 上平均约下降 30%）。 在包含 59K 个样本的完整数据池上训练——它是 s1K 的超集——相比我们选择的 1K 样本并未带来显著收益。
This highlights the importance of careful data selection and echoes prior findings for instruction tuning. For (b), we define desiderata for test-time scaling methods to compare different approaches. Budget forcing leads to the best scaling as it has perfect controllability with a clear positive slope leading to strong performance.
这凸显了精细数据选择的重要性，也呼应了此前指令微调研究的发现。 对于 (b)，我们定义了测试时扩展方法应具备的性质，以便比较不同方法。 预算强制带来了最佳扩展效果，因为它具有完全可控性和清晰的正向斜率，进而实现了较强的性能。
In summary, our contributions are: We develop simple methods for creating a sample-efficient reasoning dataset (Section 2) and test-time scaling (Section 3); Based on these we build s1-32B which is competitive with o1-preview (Section 4); We ablate subtleties of data (Section 5.1) and test-time scaling (Section 5.2). We end with a discussion to motivate future work on simple reasoning (Section 6). Our code, model, and data are open-source at https://github.com/simplescaling/s1.
2. Reasoning data curation to create s1K
In this section, we describe our process for creating a large dataset first in Section 2.1 and then filtering it down to s1K in Section 2.2.
2.1. Initial collection of 59K samples
We collect an initial 59,029 questions from 16 sources following three guiding principles. Quality: Datasets should be high-quality; we always inspect samples and ignore datasets with, e.g., poor formatting; Difficulty: Datasets should be challenging and require significant reasoning effort; Diversity: Datasets should stem from various fields to cover different reasoning tasks. We collect datasets of two categories:
我们遵循三个指导原则，从 16 个来源初步收集了 59,029 道问题。 质量：数据集应当具有高质量；我们始终检查样本，并排除格式不佳等存在问题的数据集；难度：数据集应具有挑战性，需要付出大量推理工作；多样性：数据集应来自不同领域，以覆盖不同的推理任务。 我们收集了两类数据集：
Curation of existing datasets Our largest source is NuminaMATH with 30,660 mathematical problems from online websites. We also include historical AIME problems (1983-2021). To enhance diversity, we add OlympicArena with 4,250 questions spanning Astronomy, Biology, Chemistry, Computer Science, Geography, Mathematics, and Physics from various Olympiads. OmniMath adds 4,238 competition-level mathematics problems. We also include 2,385 problems from AGIEval, which features questions from standardized tests like SAT and LSAT, covering English, Law, and Logic. We refer to Table 7 in Appendix C for our other sources.
New datasets in quantitative reasoning To complement these existing datasets, we create two original datasets. s1-prob consists of 182 questions from the probability section of Stanford University's Statistics Department's PhD Qualifying Exams (https://statistics.stanford.edu), accompanied by handwritten solutions that cover difficult proofs. The probability qualifying exam is held yearly and requires professional-level mathematical problem-solving. s1-teasers comprises 23 challenging brain-teasers commonly used in interview questions for quantitative trading positions. Each sample consists of a problem and solution taken from PuzzledQuant (https://www.puzzledquant.com/). We only take examples with the highest difficulty level ("Hard").
新的定量推理数据集 为补充这些已有数据集，我们创建了两个原创数据集。 s1-prob 包含斯坦福大学统计系博士资格考试（https://statistics.stanford.edu）概率部分的 182 道题目，并配有涵盖高难度证明的手写解答。 概率资格考试每年举行一次，要求具备专业水平的数学解题能力。 s1-teasers 包含 23 道具有挑战性的脑筋急转弯题，这类题目常用于量化交易岗位的面试。 每个样本均包含来自 PuzzledQuant（https://www.puzzledquant.com/）的一道问题及其解答。 我们仅选取最高难度级别（“Hard”）的样本。
For each question, we generate a reasoning trace and solution using the Google Gemini Flash Thinking API extracting its reasoning trace and response. This yields 59K triplets of a question, generated reasoning trace, and generated solution. Examples from our dataset are in Appendix D.2. We decontaminate all samples against our evaluation questions (MATH500, GPQA Diamond, AIME24; Appendix C.5) using 8-grams and deduplicate the data.
2.2. Final selection of 1K samples
We could directly train on our pool of 59K questions, however, our goal is to find the simplest approach with minimal resources. Thus, we go through three stages of filtering to arrive at a minimal set of 1,000 samples relying on our three guiding data principles: Quality, Difficulty, and Diversity.
我们可以直接在包含 59K 道问题的数据池上训练，但我们的目标是找到资源需求最少的 最简单 方法。 因此，我们依据质量、难度和多样性这三个数据指导原则，经过三轮筛选，得到一个仅含 1,000 个样本的最小集合。
Quality We first remove any questions where we ran into any API errors reducing our dataset to 54,116 samples. Next, we filter out low-quality examples by checking if they contain any string patterns with formatting issues, such as ASCII art diagrams, non-existent image references, or inconsistent question numbering reducing our dataset to 51,581 examples. From this pool, we identify 384 samples for our final 1,000 samples from datasets that we perceive as high-quality and not in need of further filtering (see Appendix C.4 for details).
质量 首先，我们移除所有遇到 API 错误的问题，使数据集缩减至 54,116 个样本。 接着，我们检查样本是否包含存在格式问题的字符串模式，例如 ASCII 艺术图、指向不存在图像的引用或不一致的问题编号，以过滤低质量样本，使数据集缩减至 51,581 个样本。 在这个数据池中，我们从认为质量较高且无需进一步筛选的数据集中确定了 384 个样本，纳入最终的 1,000 个样本（详情见 附录 C.4）。
Difficulty For difficulty, we use two indicators: model performance and reasoning trace length. We evaluate two models on each question: Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct, with correctness assessed by Claude 3.5 Sonnet comparing each attempt against the reference solution (see Appendix C.3 for the grading protocol). We measure the token length of each reasoning trace to indicate problem difficulty using the Qwen2.5 tokenizer. This relies on the assumption that more difficult problems require more thinking tokens.
难度 对于难度，我们使用两个指标：模型表现和推理轨迹长度。 我们在每道问题上评估两个模型：Qwen2.5-7B-Instruct 和 Qwen2.5-32B-Instruct，并由 Claude 3.5 Sonnet 将每次作答与参考解答进行比较，以评判正确性（评分协议见 附录 C.3）。 我们使用 Qwen2.5 的 tokenizer 测量每条推理轨迹的 token 长度，以表征问题难度。 这一做法基于一个假设：越困难的问题需要越多的思考 token。
Based on the grading, we remove questions that either Qwen2.5-7B-Instruct or Qwen2.5-32B-Instruct can solve correctly and thus may be too easy. By using two models we reduce the likelihood of an easy sample slipping through our filtering due to a rare mistake on an easy question of one of the models. This brings our total samples down to 24,496, setting the stage for the next round of subsampling based on diversity. While filtering with these two models may be optimized for our setup as we will also use Qwen2.5-32B-Instruct as our model to finetune, the idea of model-based filtering generalizes to other setups.
根据评分结果，我们移除 Qwen2.5-7B-Instruct 或 Qwen2.5-32B-Instruct 中任一模型能够正确解答、因而可能过于简单的问题。 同时使用两个模型，可以降低某个模型在简单问题上偶然出错、导致简单样本漏过筛选的可能性。 这将样本总数缩减至 24,496，为下一轮基于多样性的子采样做好准备。 由于我们也将使用 Qwen2.5-32B-Instruct 作为微调对象，这两个模型的筛选方式可能更适合我们的实验设置，但基于模型进行筛选的思路可以推广到其他设置。
Diversity To quantify diversity, we classify questions into domains using Claude 3.5 Sonnet based on the Mathematics Subject Classification (MSC) system (e.g., geometry, combinatorics, etc.) from the American Mathematical Society. The taxonomy focuses on topics in mathematics but also includes other sciences such as biology, physics, and economics. To select our final examples from the pool of 24,496 questions, we first choose one domain uniformly at random. Then, we sample one problem from this domain according to a distribution that favors longer reasoning traces (see Appendix C.4 for details) as motivated in Difficulty. We repeat this process until we have 1,000 total samples spanning 50 domains.
多样性 为量化多样性，我们使用 Claude 3.5 Sonnet，依据美国数学学会的数学主题分类（MSC）系统，将问题划分到不同领域，例如几何、组合数学等。 该分类体系主要关注数学主题，但也包括生物学、物理学和经济学等其他学科。 为了从 24,496 道问题中选出最终样本，我们先均匀随机地选择一个领域。 然后，依据“难度”部分的动机，按照偏向较长推理轨迹的分布，从该领域中抽取一道问题（详情见 附录 C.4）。 我们重复这一过程，直到获得覆盖 50 个领域的共 1,000 个样本。
In Section 5.1, we will show that using our three criteria in combination is important, as only relying on quality, diversity, or difficulty in isolation leads to worse datasets. Some distilled generations are incorrect, which we allow in our data as we focus on capturing the reasoning process rather than entirely correct solutions. Our grader (Appendix C.3) deems 53.6% correct in s1K and 63.0% in our follow-up s1K-1.1 (see Appendix A).
3. Test-time scaling
3.1. Method
We classify test-time scaling methods into 1) Sequential, where later computations depend on earlier ones (e.g., a long reasoning trace), and 2) Parallel, where computations run independently (e.g., majority voting). We focus on sequential scaling as intuitively we believe it should scale better, since later computations can build on intermediate results, allowing for deeper reasoning and iterative refinement. We propose new sequential scaling methods and ways to benchmark them.
我们将测试时扩展方法分为两类：1）串行，后续计算依赖于先前计算，例如较长的推理轨迹；2）并行，各次计算独立进行，例如多数投票。 我们重点研究串行扩展，因为直觉上认为它应该具有更好的扩展效果：后续计算可以利用中间结果，从而支持更深入的推理和迭代改进。 我们提出了新的串行扩展方法，以及用于评测这些方法的方式。
Budget forcing We propose a simple decoding-time intervention by forcing a maximum and/or minimum number of thinking tokens. Specifically, we enforce a maximum token count by simply appending the end-of-thinking token delimiter and optionally “Final Answer:” to early exit the thinking stage and make the model provide its current best answer. To enforce a minimum, we suppress the generation of the end-of-thinking token delimiter and optionally append the string “Wait” to the model's current reasoning trace to encourage the model to reflect on its current generation. Figure 3 contains an example of how this simple approach can lead the model to arrive at a better answer.
预算强制 我们提出一种简单的解码时干预方式，强制规定思考 token 数量的上限和／或下限。 具体而言，我们通过直接追加思考结束 token 分隔符，并可选择追加“Final Answer:”，来强制执行 token 数上限，从而提前退出思考阶段，让模型给出当前最佳答案。 为了强制执行下限，我们抑制思考结束 token 分隔符的生成，并可选择在模型当前推理轨迹之后追加字符串“Wait”，鼓励模型反思当前生成的内容。 图3 展示了这种简单方法如何使模型得到更好答案的一个例子。
Baselines We benchmark budget forcing with: (I) Conditional length-control methods, which rely on telling the model in the prompt how long it should generate for. We group them by granularity into (a) Token-conditional control: We specify an upper bound of thinking tokens in the prompt; (b) Step-conditional control: We specify an upper bound of thinking steps, where each step is around 100 tokens; (c) Class-conditional control: We write two generic prompts that tell the model to either think for a short or long amount of time (see Appendix E.1 for details). (II) Rejection sampling, which samples until a generation fits a predetermined compute budget. This oracle captures the posterior over responses conditioned on its length.
基线 我们将预算强制与以下方法进行比较：(I) 条件式长度控制方法，它们依赖于在提示中告诉模型应生成多长的内容。 我们按粒度将其分为：(a) token 条件控制，在提示中指定思考 token 数的上限；(b) 步骤条件控制，指定思考步数的上限，其中每步约含 100 个 token；(c) 类别条件控制，编写两个通用提示，分别要求模型思考较短或较长的时间（详情见 附录 E.1）。 (II) 拒绝采样，不断采样，直到生成结果满足预先设定的计算预算。 这一理想化方法刻画了以回复长度为条件的回复后验分布。
3.2. Metrics
We establish a set of desiderata as evaluation metrics to measure test-time scaling across methods. Importantly, we do not only care about the accuracy a method can achieve but also its controllability and test-time scaling slope. For each method we consider, we run a set of evaluations
我们确立了一组期望性质作为评估指标，用于衡量不同方法的测试时扩展能力。 需要强调的是，我们不仅关心一种方法能够达到的准确率，也关心其可控性和测试时扩展斜率。 对于每种方法，我们在一个固定基准（例如 AIME24）上改变测试时计算量，运行一组评估
where
其中，
Performance is simply the maximum performance the method achieves on the benchmark. A method with monotonically increasing scaling achieves 100% performance on any benchmark in the limit. However, the methods we investigate eventually flatten out or further scaling fails due to control or context window limitations.
Performance 就是该方法在基准上达到的最高性能。 对于扩展效果单调递增的方法，在极限情况下，它在任何基准上的性能都能达到 100%。 然而，我们研究的方法最终都会趋于平缓，或者因可控性或上下文窗口的限制而无法继续扩展。
4. Results
4.1. Setup
Training We perform supervised finetuning on Qwen2.5-32B-Instruct using s1K to obtain our model s1-32B using basic hyperparameters outlined in Appendix D. Finetuning took 26 minutes on 16 NVIDIA H100 GPUs with PyTorch FSDP.
训练 我们使用 s1K，按照 附录 D 所列的基本超参数，对 Qwen2.5-32B-Instruct 进行监督微调，得到模型 s1-32B。 借助 PyTorch FSDP，微调在 16 张 NVIDIA H100 GPU 上耗时 26 分钟。
Evaluation We select three representative reasoning benchmarks widely used in the field: AIME24 has 30 problems that were used in the 2024 American Invitational Mathematics Examination (AIME) held from January 31 – February 1, 2024. AIME tests mathematical problem-solving with arithmetic, algebra, counting, geometry, number theory, probability, and other secondary school math topics. High-scoring high school students in the test are invited to participate in the United States of America Mathematics Olympiad (USAMO). All AIME answers are integers ranging from
评估 我们选择了领域内广泛使用的三个代表性推理基准：AIME24 包含 30 道题目，来自 2024 年 1 月 31 日至 2 月 1 日举行的美国数学邀请赛（AIME）。 AIME 考查数学解题能力，内容包括算术、代数、计数、几何、数论、概率，以及其他中学数学主题。 在该考试中取得高分的高中生，会受邀参加美国数学奥林匹克竞赛（USAMO）。 AIME 的所有答案均为
We evaluate on the same 500 samples selected by OpenAI in prior work. GPQA Diamond consists of 198 PhD-level science questions from Biology, Chemistry and Physics. Experts with PhDs in the corresponding domains only achieved 69.7% on GPQA Diamond. When we write “GPQA” in the context of evaluation in this work, we always refer to the Diamond subset. We build on the “lm-evaluation-harness” framework. Unless otherwise specified, we evaluate with a temperature of 0 (greedy) and measure accuracy (equivalent to pass@1).
我们使用 OpenAI 在此前工作中选取的同一组 500 个样本进行评估。 GPQA Diamond 包含 198 道博士级科学问题，涵盖生物学、化学和物理学。 拥有相应领域博士学位的专家在 GPQA Diamond 上也仅取得了 69.7% 的成绩。 本文在评估语境中提到“GPQA”时，始终指 Diamond 子集。 我们基于“lm-evaluation-harness”框架开展评估。 除非另有说明，我们采用温度为 0 的贪心解码进行评估，并衡量准确率（等价于 pass@1）。
Other models We benchmark s1-32B against: OpenAI o1 series, closed-source models that popularized test-time scaling; DeepSeek r1 series, open-weight reasoning models with up to o1-level performance; Qwen's QwQ-32B-preview, a 32B open-weight reasoning model without disclosed methodology; Sky-T1-32B-Preview and Bespoke-32B, open models with open reasoning data distilled from QwQ-32B-preview and r1; Google Gemini 2.0 Flash Thinking Experimental, the API that we distill from. As it has no official evaluation scores, we use the Gemini API to benchmark it ourselves. However, the “recitation error” of the Gemini API makes evaluation challenging. We circumvent this, by manually inserting all 30 AIME24 questions in its web interface where the error does not appear. However, we leave out MATH500 (500 questions) and GPQA Diamond (198 questions), thus they are N.A. in Table 1. Our model, s1-32B, is fully open including weights, reasoning data, and code.
其他模型 我们将 s1-32B 与以下模型进行比较：推动测试时扩展流行起来的闭源模型 OpenAI o1 系列；性能最高可达 o1 水平的开放权重推理模型 DeepSeek r1 系列；Qwen 的 QwQ-32B-preview，一个未公开方法的 32B 开放权重推理模型；Sky-T1-32B-Preview 和 Bespoke-32B，它们是开放模型，并公开了从 QwQ-32B-preview 和 r1 蒸馏的推理数据；以及我们用于蒸馏的 API Google Gemini 2.0 Flash Thinking Experimental。 由于它没有官方评估分数，我们使用 Gemini API 自行对其进行评测。 然而，Gemini API 的“recitation error”使评估变得困难。 我们通过在不会出现这一错误的网页界面中，手动输入全部 30 道 AIME24 问题来绕过该问题。 不过，我们没有评估 MATH500（500 道问题）和 GPQA Diamond（198 道问题），因此它们在 表1 中标为 N.A.。 我们的模型 s1-32B 完全开放，包括权重、推理数据和代码。
| Model | # ex. | AIME 2024 | MATH 500 | GPQA Diamond |
|---|---|---|---|---|
| API only | ||||
| o1-preview | N.A. | 44.6 | 85.5 | 73.3 |
| o1-mini | N.A. | 70.0 | 90.0 | 60.0 |
| o1 | N.A. | 74.4 | 94.8 | 77.3 |
| Gemini 2.0 Flash Think. | N.A. | 60.0 | N.A. | N.A. |
| Open Weights | ||||
| Qwen2.5-32B-Instruct | N.A. | 26.7 | 84.0 | 49.0 |
| QwQ-32B | N.A. | 50.0 | 90.6 | 54.5 |
| r1 | ≫800K | 79.8 | 97.3 | 71.5 |
| r1-distill | 800K | 72.6 | 94.3 | 62.1 |
| Open Weights and Open Data | ||||
| Sky-T1 | 17K | 43.3 | 82.4 | 56.8 |
| Bespoke-32B | 17K | 63.3 | 93.0 | 58.1 |
| s1 w/o BF | 1K | 50.0 | 92.6 | 56.6 |
| s1-32B | 1K | 56.7 | 93.0 | 59.6 |
4.2. Performance
Test-time scaling Figure 1 shows the performance of s1-32B with budget forcing scales with more test-time compute. In Figure 4 (left), we expand the plot from Figure 1 (middle) showing that while we can improve AIME24 performance using our budget forcing technique (Section 3) and more test-time compute it does eventually flatten out at six times. Suppressing the end-of-thinking token delimiter too often can lead the model into repetitive loops instead of continued reasoning. In Figure 4 (right), we show that after training Qwen2.5-32B-Instruct on our 1,000 samples to produce s1-32B and equipping it with the simple budget forcing technique, it operates in a different scaling paradigm. Scaling test-time compute on the base model via majority voting cannot catch up with the performance of s1-32B which validates our intuition from Section 3 that sequential scaling is more effective than parallel. We provide example generations of s1-32B in Figure 5.
测试时扩展 图1 表明，配备预算强制的 s1-32B 的性能会随测试时计算的增加而提升。 在 图4（左）中，我们扩展了 图1（中）的曲线，表明尽管可以借助预算强制技术（第3节）和更多测试时计算提升 AIME24 性能，但在干预六次时，性能最终仍会趋于平缓。 过于频繁地抑制思考结束 token 分隔符，可能使模型陷入重复循环，而不是继续推理。 在 图4（右）中，我们展示了：使用这 1,000 个样本训练 Qwen2.5-32B-Instruct 得到 s1-32B，再为其配备简单的预算强制技术之后，模型就会以不同的扩展范式运行。 通过多数投票扩展基础模型的测试时计算，无法赶上 s1-32B 的性能，这验证了 第3节 中串行扩展比并行扩展更有效的直觉。 我们在 图5 中给出了 s1-32B 的生成示例。
Sample-efficiency In Figure 2 (right) and Table 1 we compare s1-32B with other models. We find that s1-32B is the most sample-efficient open data reasoning model. It performs significantly better than our base model (Qwen2.5-32B-Instruct) despite just training it on an additional 1,000 samples. The concurrently released r1-32B shows stronger performance than s1-32B while also only using SFT.
样本效率 我们在 图2（右）和 表1 中，将 s1-32B 与其他模型进行了比较。 我们发现，s1-32B 是公开数据的推理模型中样本效率最高的。 尽管仅额外使用 1,000 个样本训练，它的性能仍显著优于基础模型 Qwen2.5-32B-Instruct。 同期发布的 r1-32B 同样只使用 SFT，但展现出比 s1-32B 更强的性能。
However, it is trained on 800
不过，它使用的推理训练样本数是我们的 800 倍。 能否仅用 1,000 个样本达到它的性能，仍是一个开放问题。 最后，我们的模型在 AIME24 上几乎与 Gemini 2.0 Thinking 持平。 由于 s1-32B 的数据蒸馏自 Gemini 2.0，这表明我们的蒸馏过程很可能是有效的。
5. Ablations
5.1. Data Quantity, Diversity, and Difficulty
| Model | AIME 2024 | MATH 500 | GPQA Diamond |
|---|---|---|---|
| 1K-random | 36.7[−26.7%, −3.3%] | 90.6[−4.8%, 0.0%] | 52.0[−12.6%, 2.5%] |
| 1K-diverse | 26.7[−40.0%, −10.0%] | 91.2[−4.0%, 0.2%] | 54.6[−10.1%, 5.1%] |
| 1K-longest | 33.3[−36.7%, 0.0%] | 90.4[−5.0%, −0.2%] | 59.6[−5.1%, 10.1%] |
| 59K-full | 53.3[−13.3%, 20.0%] | 92.8[−2.6%, 2.2%] | 58.1[−6.6%, 8.6%] |
| s1K | 50.0 | 93.0 | 57.6 |
In Section 2 we outlined our three guiding principles in curating s1K: Quality, Difficulty, and Diversity. Here we test the importance of combining them and the overall efficacy of our selection. Only Quality (1K-random): After obtaining our high-quality reasoning chains from Gemini, we select 1,000 samples at random; not relying on our difficulty and diversity filtering at all. Table 2 shows this approach performs much worse than s1K across all benchmarks. Only Diversity (1K-diverse): For this dataset, we sample uniformly across domains to maximize diversity disregarding any notion of difficulty.
在 第2节 中，我们概述了整理 s1K 的三个指导原则：质量、难度和多样性。 这里，我们检验将三者结合的重要性，以及数据选择方案的整体有效性。 仅考虑质量（1K-random）：从 Gemini 获得高质量推理链后，我们随机选取 1,000 个样本，完全不采用难度和多样性筛选。 表2 表明，这种方法在所有基准上的表现均明显低于 s1K。 仅考虑多样性（1K-diverse）：对于这个数据集，我们在各领域之间均匀采样以最大化多样性，而不考虑任何难度因素。
This approach also leads to poor performance similar to 1K-random. Only Difficulty (1K-longest): Here we rely on one of our difficulty indicators introduced in Section 2 by selecting the 1,000 samples with the longest reasoning traces. This approach significantly boosts GPQA performance but overall still falls short of using s1K. Maximize Quantity: Finally, we compare with just training on all of our 59K samples, a superset of all the 1K-sample versions. This leads to a strong model but uses much more resources.
这种方法同样表现较差，与 1K-random 类似。 仅考虑难度（1K-longest）：这里，我们使用 第2节 提出的一个难度指标，选择推理轨迹最长的 1,000 个样本。 这种方法显著提升了 GPQA 性能，但整体上仍不及使用 s1K。 最大化数量：最后，我们与直接使用全部 59K 个样本训练的方法进行比较；这批数据是所有 1K 样本版本的超集。 这种方法能得到一个强大的模型，但消耗的资源要多得多。
To finetune on 59K samples, we use 394 H100 GPU hours while s1-32B only required 7 H100 GPU hours. Moreover, relying only on s1K is extremely competitive as shown in Section 2. Overall, combining all three criteria – Quality, Difficulty, Diversity – via our methodology in Section 2 is key for sample-efficient reasoning training.
5.2. Test-time scaling methods
| Method | Control | Scaling | Performance | |𝒜| |
|---|---|---|---|---|
| BF | 100% | 15 | 56.7 | 5 |
| TCC | 40% | −24 | 40.0 | 5 |
| TCC + BF | 100% | 13 | 40.0 | 5 |
| SCC | 60% | 3 | 36.7 | 5 |
| SCC + BF | 100% | 6 | 36.7 | 5 |
| CCC | 50% | 25 | 36.7 | 2 |
| RS | 100% | −35 | 40.0 | 5 |
Budget forcing In Table 3 we compare the test-time scaling methods we have introduced in Section 3. Overall, we find that budget forcing provides perfect control, good scaling, and leads to our best AIME24 score. Thus, this is the method we use for s1-32B in Figure 1 and in Section 4. In Table 4, we compare different strings for extrapolating performance. We find that “Wait” generally gives the best performance.
Class-conditional control We provide benchmark scores for this method in Appendix E.1 and summarize three findings here: (1) Token-conditional control fails without budget forcing, as our model cannot reliably count tokens - even when trained to do so. (2) Under step-conditional control, the model generates a similar total number of tokens when given different step targets, as the model goes from few steps with many tokens per step, to many steps with few tokens in each step. Thus, the model learns to hack its way around the compute constraint making the controllability of this method mediocre. (3) Class-conditional control can work - telling a model to simply think longer can increase its test-time compute and performance, which leads good scaling in Table 3.
类别条件控制 我们在 附录 E.1 给出了该方法的基准分数，这里总结三个发现：(1) 在没有预算强制的情况下，token 条件控制会失效，因为即使专门接受过训练，我们的模型也无法可靠地统计 token 数。 (2) 在步骤条件控制下，面对不同的目标步数，模型生成的 token 总数相近，因为模型会从“步数少、每步 token 多”转变为“步数多、每步 token 少”。 因此，模型学会了钻计算约束的空子，使得这种方法的可控性一般。 (3) 类别条件控制是可行的：仅仅要求模型思考更久，就能增加测试时计算量并提升性能，因此在 表3 中表现出良好的扩展效果。
| Model | AIME 2024 | MATH 500 | GPQA Diamond |
|---|---|---|---|
| No extrapolation | 50.0 | 93.0 | 57.6 |
| 2x without string | 50.0 | 90.2 | 55.1 |
| 2x “Alternatively” | 50.0 | 92.2 | 59.6 |
| 2x “Hmm” | 50.0 | 93.0 | 59.6 |
| 2x “Wait” | 53.3 | 93.0 | 59.6 |
Rejection sampling Surprisingly, we find that simply sampling until the generation fits a specific length leads to an inverse scaling trend as depicted in Figure 6. In Appendix E.2 we inspect a question, which was answered correctly by the model when rejection sampling for
拒绝采样 令人意外的是，我们发现，仅仅不断采样直到生成结果满足特定长度，就会产生 图6 所示的反向扩展趋势。 在 附录 E.2 中，我们检查了一道题：当拒绝采样要求不超过 4000 个 token 时，模型能够正确作答，但在不超过 8000 个 token 的设置下却不能。 在不超过 4000 个 token 的设置下，模型会直接采用正确方法，而在不超过 8000 个 token 的设置下，模型会进行大量回溯。 我们猜想存在如下相关性：较短的生成结果往往对应模型从一开始就走在正确方向上的情况，而较长的结果往往对应模型犯错后不断回溯或自我质疑的情况。 这使得拒绝采样中的较长样本往往是错误的，从而产生反向扩展趋势。
6. Discussion and related work
6.1. Sample-efficient reasoning
Models There are a number of concurrent efforts to build models that replicate the performance of o1. For example, DeepSeek-r1 and k1.5 are built with reinforcement learning methods, while others rely on SFT using tens of thousands of distilled examples. We show that SFT on only 1,000 examples suffices to build a competitive reasoning model matching o1-preview and produces a model that lies on the pareto frontier (Figure 2). Further, we introduce budget forcing which combined with our reasoning model leads to the first reproduction of OpenAI's test-time scaling curves. Why does supervised finetuning on just 1,000 samples lead to such performance gains?
模型 有多项同期工作致力于构建能够复现 o1 性能的模型。 例如，DeepSeek-r1 和 k1.5 使用强化学习方法构建，而其他工作则依赖于使用数万个蒸馏样本进行 SFT。 我们表明，仅在 1,000 个样本上进行 SFT，就足以构建一个能够与 o1-preview 匹敌的推理模型，并使其位于帕累托前沿（图2）。 此外，我们提出的预算强制与推理模型结合后，首次复现了 OpenAI 的测试时扩展曲线。 为什么仅在 1,000 个样本上进行监督微调，就能带来如此大的性能提升？
We hypothesize that the model is already exposed to large amounts of reasoning data during pretraining which spans trillions of tokens. Thus, the ability to perform reasoning is already present in our model. Our sample-efficient finetuning stage just activates it and we scale it further at test time with budget forcing. This is similar to the "Superficial Alignment Hypothesis" presented in LIMA, where the authors find that 1,000 examples can be sufficient to align a model to adhere to user preferences.
我们猜想，模型在包含数万亿 token 的预训练中，已经接触过大量推理数据。 因此，我们的模型本来就具备推理能力。 高样本效率的微调阶段只是激活了这种能力，而我们再通过预算强制在测试时进一步扩展它。 这与 LIMA 提出的“表层对齐假说”类似：该研究发现，1,000 个样本就可能足以将模型对齐到用户偏好。
Benchmarks and methods To evaluate and push the limits of these models, increasingly challenging benchmarks have been introduced, such as Olympiad-level science competitions and others. To enhance models’ performance on reasoning-related tasks, researchers have pursued several strategies: Prior works have explored continuing training language models on specialized corpora related to mathematics and science, sometimes even synthetically generated data. Others have developed training methodologies specifically aimed at reasoning performance. Another significant line of work focuses on prompting-based methods to elicit and improve reasoning abilities, including methods like Chain-of-Thought prompting. These combined efforts aim to advance the reasoning ability of language models, enabling them to handle more complex and abstract tasks effectively.
基准与方法 为评估这些模型并探索其能力极限，研究者提出了越来越有挑战性的基准，例如奥林匹克级科学竞赛以及其他基准。 为了提升模型在推理相关任务上的表现，研究者采用了多种策略：已有工作探索了在数学和科学相关的专门语料上继续训练语言模型，有时甚至使用合成生成的数据。 另一些工作开发了专门针对推理性能的训练方法。 另一条重要研究路线关注基于提示的方法，以激发并提升推理能力，其中包括思维链提示等方法。 这些努力共同推动语言模型推理能力的发展，使其能够有效处理更复杂、更抽象的任务。
6.2. Test-time scaling
Methods As we introduce in Section 3, we differentiate two methods to scale test-time compute: parallel and sequential. The former relies on multiple solution attempts generated in parallel and selecting the best outcome via specific criteria. These criteria include choosing the most frequent response for majority voting or the best response based on an external reward for Best-of-N. Unlike repeated sampling, previous sequential scaling methods let the model generate solution attempts sequentially based on previous attempts, allowing it to refine each attempt based on previous outcomes. Tree-based search methods offer a hybrid approach between sequential and parallel scaling, such as Monte-Carlo Tree Search (MCTS) and guided beam search.
方法 如 第3节 所述，我们区分了两种扩展测试时计算的方法：并行 和 串行。 前者依赖于并行生成多个候选解答，再依据特定标准选出最佳结果。 这些标准包括多数投票中选择出现频率最高的回复，以及 Best-of-N 中依据外部奖励选择最佳回复。 与重复采样不同，以往的串行扩展方法让模型依据先前尝试，依次生成新的解答，从而能够根据此前结果改进每次尝试。 基于树的搜索方法提供了串行与并行扩展之间的混合方案，例如蒙特卡洛树搜索（MCTS）和引导式束搜索。
REBASE employs a process reward model to balance exploitation and pruning during tree search. Empirically, REBASE has been shown to outperform sampling-based methods and MCTS. Reward models play a key role in these methods. They come in two variants: outcome reward models and process reward models. Outcome reward models assign a score to complete solutions and are particularly useful in Best-of-N selection, while process reward models assess individual reasoning steps and are effective in guiding tree-based search methods.
REBASE 使用过程奖励模型，在树搜索中平衡利用与剪枝。 实证结果表明，REBASE 的表现优于基于采样的方法和 MCTS。 奖励模型在这些方法中发挥着关键作用。 它们分为两类：结果奖励模型和过程奖励模型。 结果奖励模型为完整解答赋予分数，尤其适用于 Best-of-N 选择；过程奖励模型则评估各个推理步骤，能够有效引导基于树的搜索方法。
Limits to further test-time scaling We have shown that budget forcing allows extrapolating test-time compute in Section 4, e.g., improving AIME24 performance from 50% to 57%. However, it has two key limitations when scaling further: it eventually flattens out (Figure 4), and the context window of the underlying language model constrains it. Despite these constraints, our work shows test-time scaling across a wide range of accuracies (Figure 1), partly because scaling down test-time compute behaves predictably and does not suffer from these constraints.
进一步扩展测试时计算的限制 我们在 第4节 表明，预算强制能够实现测试时计算的外推，例如将 AIME24 性能从 50% 提升至 57%。 然而，进一步扩展时存在两个关键限制：性能最终会 趋于平缓（图4），并且受到基础语言模型 上下文窗口 的约束。 尽管存在这些限制，我们的工作仍在较大的准确率范围内展示了测试时扩展（图1），部分原因在于，减少测试时计算的行为是可预测的，而且不受这些限制影响。
Continuing test-time scaling will require approaches that can further extrapolate test-time compute. How can we get such extrapolation? There may be improvements to budget forcing such as rotating through different strings, not only “Wait”, or combining it with frequency penalties or higher temperature to avoid repetitive loops. An exciting direction for future work is also researching whether applying budget forcing to a reasoning model trained with reinforcement learning yields better extrapolation; or if RL allows for new ways of test-time scaling beyond budget forcing. Our work defines the right metrics (Section 3.2) – Control, Scaling, and Performance – to enable future research and progress on extrapolating test-time compute.
要继续扩展测试时计算，需要能够进一步外推测试时计算量的方法。 如何实现这样的外推？ 预算强制或许还可以改进，例如轮换不同字符串，而不仅仅使用“Wait”，或者结合频率惩罚或更高的温度来避免重复循环。 另一个令人期待的未来方向是研究：将预算强制应用于经过强化学习训练的推理模型，是否能获得更好的外推效果；或者，RL 是否能带来预算强制之外的测试时扩展新方法。 我们的工作定义了合适的指标（第3.2节）——可控性、扩展斜率和性能——以支持未来关于测试时计算外推的研究和进展。
Parallel scaling as a solution Parallel scaling offers one solution to the limits of sequential scaling, thus we augment our sequentially scaled model with two methods: (I) Majority voting: After generating
以并行扩展作为解决方案 并行扩展为串行扩展的限制提供了一种解决方案，因此，我们采用两种方法增强经过串行扩展的模型：(I) 多数投票： 生成