---

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao Z, Wang P, Zhu Q, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024.

https://github.com/deepseek-ai/DeepSeek-Math

LLMRLGRPODeepSeek清华大学北京大学

---

DeepSeekMath：推动开放语言模型数学推理能力的极限

Abstract

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH.

数学推理因其复杂且结构化的性质，对语言模型构成了重大挑战。 在本文中，我们介绍 DeepSeekMath 7B：它在 DeepSeek-Coder-Base-v1.5 7B 的基础上，使用来自 Common Crawl 的 120B 数学相关 token，并结合自然语言与代码数据继续预训练。 DeepSeekMath 7B 在不依赖外部工具包和投票技术的情况下，在竞赛级 MATH 基准上取得了令人瞩目的 51.7%，接近 Gemini-Ultra 和 GPT-4 的性能水平。 对 DeepSeekMath 7B 的 64 个样本进行自洽性评估，在 MATH 上达到 60.9%。

The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

DeepSeekMath 的数学推理能力归因于两个关键因素：第一，我们通过精心设计的数据选择流水线，发掘了公开可用 Web 数据的巨大潜力。 第二，我们提出组相对策略优化（GRPO），它是近端策略优化（PPO）的一种变体，在增强数学推理能力的同时优化 PPO 的内存使用。

图1：在不使用外部工具包和投票技术的情况下，开源模型在竞赛级 MATH 基准上的 Top-1 准确率。

1. Introduction

Large language models (LLM) have revolutionized the approach to mathematical reasoning in artificial intelligence, spurring significant advancements in both the quantitative reasoning benchmark and the geometry reasoning benchmark. Moreover, these models have proven instrumental in assisting humans in solving complex mathematical problems. However, cutting-edge models such as GPT-4 and Gemini-Ultra are not publicly available, and the currently accessible open-source models considerably trail behind in performance.

大语言模型（LLM）革新了人工智能中的数学推理方法，推动定量推理基准和几何推理基准取得了显著进展。 此外，这些模型已被证明能够有效协助人类解决复杂的数学问题。 然而，GPT-4 和 Gemini-Ultra 等前沿模型并未公开，而目前可用的开源模型在性能上明显落后。

In this study, we introduce DeepSeekMath, a domain-specific language model that significantly outperforms the mathematical capabilities of open-source models and approaches the performance level of GPT-4 on academic benchmarks. To achieve this, we create the DeepSeekMath Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. This dataset is extracted from the Common Crawl (CC) using a fastText-based classifier. In the initial iteration, the classifier is trained using instances from OpenWebMath as positive examples, while incorporating a diverse selection of other web pages to serve as negative examples.

在本研究中，我们介绍 DeepSeekMath，这是一种领域专用语言模型，其数学能力显著优于开源模型，并在学术基准上接近 GPT-4 的性能水平。 为实现这一目标，我们创建了 DeepSeekMath Corpus，这是一个由 120B 数学 token 构成的大规模高质量预训练语料库。 该数据集通过基于 fastText 的分类器从 Common Crawl（CC）中提取。 在初次迭代中，分类器以 OpenWebMath 中的样本作为正例，同时采用多样化的其他网页作为负例进行训练。

Subsequently, we employ the classifier to mine additional positive instances from the CC, which are further refined through human annotation. The classifier is then updated with this enhanced dataset to improve its performance. The evaluation results indicate that the large-scale corpus is of high quality, as our base model DeepSeekMath-Base 7B achieves 64.2% on GSM8K and 36.2% on the competition-level MATH dataset, outperforming Minerva 540B. In addition, the DeepSeekMath Corpus is multilingual, so we notice an improvement in Chinese mathematical benchmarks. We believe that our experience in mathematical data processing is a starting point for the research community, and there is significant room for improvement in the future.

随后，我们使用该分类器从 CC 中挖掘更多正例，并通过人工标注进一步精炼这些样本。 之后，我们用这个增强后的数据集更新分类器，以提升其性能。 评估结果表明，这一大规模语料库质量很高：我们的基础模型 DeepSeekMath-Base 7B 在 GSM8K 上达到 64.2%，在竞赛级 MATH 数据集上达到 36.2%，优于 Minerva 540B。 此外，DeepSeekMath Corpus 是多语言语料库，因此我们观察到模型在中文数学基准上也有所提升。 我们相信，我们的数学数据处理经验可以成为研究社区的起点，未来仍有很大的改进空间。

DeepSeekMath-Base is initialized with DeepSeek-Coder-Base-v1.5 7B, as we notice that starting from a code training model is a better choice compared to a general LLM. Furthermore, we observe the math training also improves model capability on MMLU and BBH benchmarks, indicating it does not only enhance the model's mathematical abilities but also amplifies general reasoning capabilities.

DeepSeekMath-Base 由 DeepSeek-Coder-Base-v1.5 7B 初始化，因为我们发现，与通用 LLM 相比，从经过代码训练的模型开始是更好的选择。 此外，我们观察到数学训练还提升了模型在 MMLU 和 BBH 基准上的能力，这表明它不仅增强模型的数学能力，也提高了通用推理能力。

After pre-training, we apply mathematical instruction tuning to DeepSeekMath-Base with chain-of-thought, program-of-thought, and tool-integrated reasoning data. The resulting model DeepSeekMath-Instruct 7B beats all 7B counterparts and is comparable with 70B open-source instruction-tuned models.

预训练之后，我们使用思维链、程序思维和工具集成推理数据对 DeepSeekMath-Base 进行数学指令微调。 由此得到的 DeepSeekMath-Instruct 7B 优于所有同规模 7B 模型，并可与 70B 开源指令微调模型相媲美。

Furthermore, we introduce the Group Relative Policy Optimization (GRPO), a variant reinforcement learning (RL) algorithm of Proximal Policy Optimization (PPO). GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources. By solely using a subset of English instruction tuning data, GRPO obtains a substantial improvement over the strong DeepSeekMath-Instruct, including both in-domain (GSM8K: 82.9% $\to$ 88.2%, MATH: 46.8% $\to$ 51.7%) and out-of-domain mathematical tasks (e.g., CMATH: 84.6% $\to$ 88.8%) during the reinforcement learning phase.

此外，我们提出组相对策略优化（GRPO），它是近端策略优化（PPO）的一种强化学习（RL）算法变体。 GRPO 不再使用评论家模型，而是根据组分数估计基线，从而显著减少训练资源。 仅使用英文指令微调数据的一个子集，GRPO 就在强化学习阶段相较强大的 DeepSeekMath-Instruct 取得了显著提升，既包括域内任务（GSM8K：82.9% $\to$ 88.2%，MATH：46.8% $\to$ 51.7%），也包括域外数学任务（例如 CMATH：84.6% $\to$ 88.8%）。

We also provide a unified paradigm to understand different methods, such as Rejection Sampling Fine-Tuning (RFT), Direct Preference Optimization (DPO), PPO and GRPO. Based on such a unified paradigm, we find that all these methods are conceptualized as either direct or simplified RL techniques. We also conduct extensive experiments, e.g., online v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative RL and so on, to deeply investigate the essential elements of this paradigm. At last, we explain why our RL boosts the performance of instruction-tuned models, and further summarize potential directions to achieve more effective RL based on this unified paradigm.

我们还提供了一个统一范式，用于理解拒绝采样微调（RFT）、直接偏好优化（DPO）、PPO 和 GRPO 等不同方法。 基于这一统一范式，我们发现所有这些方法都可以被理解为直接或简化的强化学习技术。 我们还开展了广泛实验，例如在线与离线训练、结果监督与过程监督、单轮与迭代强化学习等，以深入研究这一范式的基本要素。 最后，我们解释了强化学习为何能提升指令微调模型的性能，并基于这一统一范式进一步总结了实现更有效强化学习的潜在方向。

1.1 Contributions

Our contribution includes scalable math pre-training, along with the exploration and analysis of reinforcement learning.

我们的贡献包括可扩展数学预训练，以及对强化学习的探索与分析。

Math Pre-Training at Scale

大规模数学预训练

• Our research provides compelling evidence that the publicly accessible Common Crawl data contains valuable information for mathematical purposes. By implementing a meticulously designed data selection pipeline, we successfully construct the DeepSeekMath Corpus, a high-quality dataset of 120B tokens from web pages filtered for mathematical content, which is almost 7 times the size of the math web pages used by Minerva and 9 times the size of the recently released OpenWebMath.
• Our pre-trained base model DeepSeekMath-Base 7B achieves comparable performance with Minerva 540B, indicating the number of parameters is not the only key factor in mathematical reasoning capability. A smaller model pre-trained on high-quality data could achieve strong performance as well.
• We share our findings from math training experiments. Code training prior to math training improves models' ability to solve mathematical problems both with and without tool use. This offers a partial answer to the long-standing question: does code training improve reasoning abilities? We believe it does, at least for mathematical reasoning.
• Although training on arXiv papers is common, especially in many math-related papers, it brings no notable improvements on all mathematical benchmarks adopted in this paper.

• 我们的研究提供了有力证据，表明公开可用的 Common Crawl 数据包含有价值的数学信息。通过实施精心设计的数据选择流水线，我们成功构建了 DeepSeekMath Corpus：这是一个由筛选出数学内容的网页构成、包含 120B token 的高质量数据集，其规模约为 Minerva 所用数学网页的 7 倍，也是近期发布的 OpenWebMath 的 9 倍。
• 我们的预训练基础模型 DeepSeekMath-Base 7B 达到与 Minerva 540B 相当的性能，这表明参数数量并不是决定数学推理能力的唯一关键因素。在高质量数据上预训练的较小模型同样可以获得很强的性能。
• 我们分享了数学训练实验中的发现。在数学训练之前进行代码训练，可以提升模型在使用和不使用工具两种情况下解决数学问题的能力。这为一个长期存在的问题提供了部分答案：代码训练是否会提升推理能力？ 我们认为会，至少对于数学推理而言如此。
• 尽管在 arXiv 论文上训练很常见，尤其是在许多数学相关论文中，但它在本文采用的所有数学基准上都没有带来明显提升。

Exploration and Analysis of Reinforcement Learning

强化学习的探索与分析

• We introduce Group Relative Policy Optimization (GRPO), an efficient and effective reinforcement learning algorithm. GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources compared to Proximal Policy Optimization (PPO).
• We demonstrate that GRPO significantly enhances the performance of our instruction-tuned model DeepSeekMath-Instruct, by solely using the instruction-tuning data. Furthermore, we observe enhancements in the out-of-domain performance during the reinforcement learning process.
• We provide a unified paradigm to understand different methods, such as RFT, DPO, PPO, and GRPO. We also conduct extensive experiments, e.g., online v.s. offline training, outcome v.s. process supervision, single-turn v.s. iterative reinforcement learning, and so on to deeply investigate the essential elements of this paradigm.
• Based on our unified paradigm, we explore the reasons behind the effectiveness of reinforcement learning, and summarize several potential directions to achieve more effective reinforcement learning of LLMs.

• 我们提出组相对策略优化（GRPO），这是一种高效且有效的强化学习算法。GRPO 不使用评论家模型，而是根据组分数估计基线，与近端策略优化（PPO）相比显著减少训练资源。
• 我们证明，仅使用指令微调数据，GRPO 就能显著提升指令微调模型 DeepSeekMath-Instruct 的性能。此外，我们还观察到强化学习过程中域外性能得到提升。
• 我们提供了一个统一范式，用于理解 RFT、DPO、PPO 和 GRPO 等不同方法。我们还开展了在线与离线训练、结果与过程监督、单轮与迭代强化学习等广泛实验，以深入研究该范式的基本要素。
• 基于统一范式，我们探索强化学习有效的原因，并总结了若干实现更有效 LLM 强化学习的潜在方向。

1.2 Summary of Evaluations and Metrics

• English and Chinese Mathematical Reasoning: We conduct comprehensive assessments of our models on English and Chinese benchmarks, covering mathematical problems from grade-school level to college level. English benchmarks include GSM8K, MATH, SAT, OCW Courses, MMLU-STEM. Chinese benchmarks include MGSM-zh, CMATH, Gaokao-MathCloze, and Gaokao-MathQA. We evaluate models' ability to generate self-contained text solutions without tool use, and also the ability to solve problems using Python.

• 英文与中文数学推理： 我们在英文和中文基准上对模型进行全面评估，覆盖从小学到大学水平的数学问题。英文基准包括 GSM8K、MATH、SAT、OCW Courses 和 MMLU-STEM；中文基准包括 MGSM-zh、CMATH、Gaokao-MathCloze 和 Gaokao-MathQA。我们既评估模型在不使用工具的情况下生成自包含文本解答的能力，也评估其使用 Python 解决问题的能力。

• On English benchmarks, DeepSeekMath-Base is competitive with the closed-source Minerva 540B, and surpasses all open-source base models (e.g., Mistral 7B and Llemma-34B), regardless of whether they've undergone math pre-training or not, often by a significant margin. Notably, DeepSeekMath-Base is superior on Chinese benchmarks, likely because we don't follow previous works to collect English-only math pre-training data, and also include high-quality non-English ones. With mathematical instruction tuning and reinforcement learning, the resulting DeepSeekMath-Instruct and DeepSeekMath-RL demonstrate strong performance, obtaining an accuracy of over 50% on the competition-level MATH dataset for the first time within the open-source community.
• Formal Mathematics: We evaluate DeepSeekMath-Base using the informal-to-formal theorem proving task on miniF2F with Isabelle chosen to be the proof assistant. DeepSeekMath-Base demonstrates strong few-shot autoformalization performance.
• Natural Language Understanding, Reasoning, and Code: To build a comprehensive profile of models' general understanding, reasoning, and coding capabilities, we evaluate DeepSeekMath-Base on the Massive Multitask Language Understanding (MMLU) benchmark which encompasses 57 multiple-choice tasks covering diverse subjects, BIG-Bench Hard (BBH) which consists of 23 challenging tasks that mostly require multi-step reasoning to solve, as well as HumanEval and MBPP which are widely used to evaluate code language models. Math pre-training benefits both language understanding and reasoning performance.

• 在英文基准上，DeepSeekMath-Base 可与闭源 Minerva 540B 竞争，并且无论开源基础模型是否经过数学预训练，都能超过所有这类模型（例如 Mistral 7B 和 Llemma-34B），通常优势显著。值得注意的是，DeepSeekMath-Base 在中文基准上表现更好，这可能是因为我们没有像先前工作那样只收集英文数学预训练数据，还纳入了高质量的非英文数据。经过数学指令微调和强化学习后，得到的 DeepSeekMath-Instruct 与 DeepSeekMath-RL 展现出强劲性能，首次在开源社区中使竞赛级 MATH 数据集的准确率超过 50%。
• 形式数学： 我们在 miniF2F 上使用非形式化到形式化定理证明任务评估 DeepSeekMath-Base，并选择 Isabelle 作为证明助手。DeepSeekMath-Base 展现出很强的少样本自动形式化性能。
• 自然语言理解、推理与代码： 为全面刻画模型的通用理解、推理和编码能力，我们在大规模多任务语言理解（MMLU）基准、BIG-Bench Hard（BBH）、HumanEval 和 MBPP 上评估 DeepSeekMath-Base。MMLU 包含覆盖多种学科的 57 个多项选择任务，BBH 包含 23 个大多需要多步推理才能解决的高难度任务，而 HumanEval 和 MBPP 被广泛用于评估代码语言模型。数学预训练同时有益于语言理解和推理性能。

2. Math Pre-Training

2.1 Data Collection and Decontamination

In this section, we will outline the process of constructing the DeepSeekMath Corpus from Common Crawl. As depicted in Figure 2, we present an iterative pipeline that demonstrates how to systematically gather a large-scale mathematical corpus from Common Crawl, starting with a seed corpus (e.g., a small but high-quality collection of math-related dataset). It's worth noting that this approach is also applicable to other domains, such as coding.

在本节中，我们将概述从 Common Crawl 构建 DeepSeekMath Corpus 的过程。 如图2所示，我们展示了一条迭代式流水线，说明如何从种子语料库（例如规模较小但质量较高的数学相关数据集集合）出发，系统地从 Common Crawl 收集大规模数学语料库。 值得注意的是，该方法也适用于代码等其他领域。

图2：从 Common Crawl 收集数学网页的迭代式流水线。

First, we choose OpenWebMath, a collection of high-quality mathematical web texts, as our initial seed corpus. Using this corpus, we train a fastText model to recall more OpenWebMath-like mathematical web pages. Specifically, we randomly select 500,000 data points from the seed corpus as positive training examples and another 500,000 web pages from Common Crawl as negative ones. We employ an open-source library for training, configuring the vector dimension to 256, learning rate to 0.1, the maximum length of word n-gram to 3, the minimum number of word occurrences to 3, and the number of training epochs to 3.

首先，我们选择高质量数学网页文本集合 OpenWebMath 作为初始种子语料库。 我们使用该语料库训练 fastText 模型，以召回更多类似 OpenWebMath 的数学网页。 具体而言，我们从种子语料库中随机选择 500,000 个数据点作为正训练样本，并从 Common Crawl 中另选 500,000 个网页作为负样本。 我们使用开源库进行训练，将向量维度设为 256、学习率设为 0.1、词 n-gram 最大长度设为 3、词最小出现次数设为 3，并将训练轮数设为 3。

To reduce the size of the original Common Crawl, we employ URL-based deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then recall mathematical web pages from deduplicated Common Crawl with the fastText model. To filter out low-quality mathematical content, we rank the collected pages according to their scores predicted by the fastText model, and only preserve the top-ranking ones. The volume of data preserved is assessed through pre-training experiments on the top 40B, 80B, 120B, and 160B tokens. In the first iteration, we choose to keep the top 40B tokens.

为缩减原始 Common Crawl 的规模，我们采用基于 URL 的去重与近似去重技术，得到 40B 个 HTML 网页。 随后，我们使用 fastText 模型从去重后的 Common Crawl 中召回数学网页。 为过滤低质量数学内容，我们按照 fastText 模型预测的分数对收集到的网页排序，只保留排名靠前的网页。 我们分别对排名最前的 40B、80B、120B 和 160B token 进行预训练实验，以评估应保留的数据量。 在第一次迭代中，我们选择保留排名最前的 40B token。

After the first iteration of data collection, numerous mathematical web pages remain uncollected, mainly because the fastText model is trained on a set of positive examples that lacks sufficient diversity. We therefore identify additional mathematical web sources to enrich the seed corpus, so that we can optimize the fastText model. Specifically, we first organize the entire Common Crawl into disjoint domains; a domain is defined as web pages sharing the same base URL. For each domain, we calculate the percentage of web pages that are collected in the first iteration.

第一次数据收集迭代后，仍有大量数学网页未被收集，主要原因是 fastText 模型所用的正样本集合缺乏足够的多样性。 因此，我们识别更多数学网页来源以丰富种子语料库，从而优化 fastText 模型。 具体而言，我们首先将整个 Common Crawl 划分为互不重叠的域； 一个域被定义为共享同一基础 URL 的网页集合。 对于每个域，我们计算其中在第一次迭代中已被收集的网页比例。

Domains where over 10% of the web pages have been collected are classified as math-related (e.g., mathoverflow.net). Subsequently, we manually annotate the URLs associated with mathematical content within these identified domains (e.g., mathoverflow.net/questions). Web pages linked to these URLs, yet uncollected, will be added to the seed corpus. This approach enables us to gather more positive examples, thereby training an improved fastText model capable of recalling more mathematical data in the subsequent iteration.

已收集网页比例超过 10% 的域会被归类为数学相关域（例如 mathoverflow.net）。 随后，我们人工标注这些已识别域中与数学内容相关的 URL（例如 mathoverflow.net/questions）。 链接到这些 URL 但尚未被收集的网页将被加入种子语料库。 该方法使我们能够收集更多正样本，进而训练出改进的 fastText 模型，在下一次迭代中召回更多数学数据。

After four iterations of data collection, we end up with 35.5M mathematical web pages, totaling 120B tokens. In the fourth iteration, we notice that nearly 98% of the data has already been collected in the third iteration, so we decide to cease data collection.

经过四次数据收集迭代，我们最终得到 35.5M 个数学网页，共计 120B token。 在第四次迭代中，我们注意到近 98% 的数据已在第三次迭代中收集，因此决定停止数据收集。

To avoid benchmark contamination, we filter out web pages containing questions or answers from English mathematical benchmarks such as GSM8K and MATH and Chinese benchmarks such as CMATH and AGIEval. The filtering criteria are as follows: any text segment containing a 10-gram string that matches exactly with any sub-string from the evaluation benchmarks is removed from our math training corpus. For benchmark texts that are shorter than 10 grams but have at least 3 grams, we employ exact matching to filter out contaminated web pages.

为避免基准污染，我们过滤掉包含英文数学基准 GSM8K、MATH 以及中文基准 CMATH、AGIEval 中问题或答案的网页。 过滤标准如下：任何包含与评估基准中任一子字符串完全匹配的 10-gram 字符串的文本片段，都会从数学训练语料库中移除。 对于短于 10 gram 但至少包含 3 gram 的基准文本，我们采用精确匹配来过滤受污染网页。

2.2 Validating the Quality of the DeepSeekMath Corpus

We run pre-training experiments to investigate how the DeepSeekMath Corpus is compared with the recently released math-training corpora:

我们通过预训练实验，研究 DeepSeekMath Corpus 与近期发布的数学训练语料库相比表现如何：

• MathPile: a multi-source corpus (8.9B tokens) aggregated from textbooks, Wikipedia, ProofWiki, CommonCrawl, StackExchange, and arXiv, with the majority (over 85%) sourced from arXiv;
• OpenWebMath: CommonCrawl data filtered for mathematical content, totaling 13.6B tokens;
• Proof-Pile-2: a mathematical corpus consisting of OpenWebMath, AlgebraicStack (10.3B tokens of mathematical code), and arXiv papers (28.0B tokens). When experimenting on Proof-Pile-2, we use an arXiv:Web:Code ratio of 2:4:1.

• MathPile： 一个多源语料库（8.9B token），汇集自教科书、Wikipedia、ProofWiki、CommonCrawl、StackExchange 和 arXiv，其中大部分（超过 85%）来自 arXiv；
• OpenWebMath： 从 CommonCrawl 中筛选出的数学内容，共计 13.6B token；
• Proof-Pile-2： 一个由 OpenWebMath、AlgebraicStack（10.3B 数学代码 token）和 arXiv 论文（28.0B token）组成的数学语料库。在 Proof-Pile-2 实验中，我们采用 2:4:1 的 arXiv、Web 与 Code 比例。

2.2.1 Training Setting

We apply math training to a general pre-trained language model with 1.3B parameters, which shares the same framework as the DeepSeek LLMs, denoted as DeepSeek-LLM 1.3B. We separately train a model on each mathematical corpus for 150B tokens. All experiments are conducted using the efficient and light-weight HAI-LLM training framework.

我们对一个具有 1.3B 参数的通用预训练语言模型进行数学训练，该模型与 DeepSeek LLM 采用相同框架，记为 DeepSeek-LLM 1.3B。 我们分别在每个数学语料库上训练一个模型，每个模型训练 150B token。 所有实验均使用高效轻量的 HAI-LLM 训练框架完成。

Following the training practice of DeepSeek LLMs, we use the AdamW optimizer with ${\beta }_1=0.9$, ${\beta }_2=0.95$, and $\mathrm{weight}\mathrm{\_}\mathrm{decay}=0.1$, along with a multi-step learning rate schedule where the learning rate reaches the peak after 2,000 warmup steps, decreases to its 31.6% after 80% of the training process, and further decreases to 10.0% of the peak after 90% of the training process. We set the maximum value of learning rate to 5.3e-4, and use a batch size of 4M tokens with a 4K context length.

沿用 DeepSeek LLM 的训练实践，我们使用 AdamW 优化器，其中 ${\beta }_1=0.9$、${\beta }_2=0.95$、$\mathrm{weight}\mathrm{\_}\mathrm{decay}=0.1$；同时采用多阶段学习率调度：学习率在 2,000 个预热步骤后达到峰值，在训练进程完成 80% 后降至峰值的 31.6%，在完成 90% 后进一步降至峰值的 10.0%。 我们将最大学习率设为 5.3e-4，并在 4K 上下文长度下使用 4M token 的批量大小。

表1：DeepSeek-LLM 1.3B 在不同数学语料库上训练后的性能，使用少样本思维链提示进行评估。语料库规模使用词表大小为 100K 的分词器计算。

Math Corpus	Size	English Benchmarks					Chinese Benchmarks
		GSM8K	MATH	OCW	SAT	MMLU STEM	CMATH	Gaokao MathCloze	Gaokao MathQA
No Math Training	N/A	2.9%	3.0%	2.9%	15.6%	19.5%	12.3%	0.8%	17.9%
MathPile	8.9B	2.7%	3.3%	2.2%	12.5%	15.7%	1.2%	0.0%	2.8%
OpenWebMath	13.6B	11.5%	8.9%	3.7%	31.3%	29.6%	16.8%	0.0%	14.2%
Proof-Pile-2	51.9B	14.3%	11.2%	3.7%	43.8%	29.2%	19.9%	5.1%	11.7%
DeepSeekMath Corpus	120.2B	23.8%	13.6%	4.8%	56.3%	33.1%	41.5%	5.9%	23.6%

图3：DeepSeek-LLM 1.3B 在不同数学语料库上训练时的基准曲线。

2.2.2 Evaluation Results

The DeepSeekMath Corpus is of high quality, covers multilingual mathematical content, and is the largest in size.

DeepSeekMath Corpus 质量高、涵盖多语言数学内容，并且规模最大。

• High-quality: We evaluate downstream performance on 8 mathematical benchmarks using few-shot chain-of-thought prompting. As shown in Table 1, there is a clear performance lead of the model trained on the DeepSeekMath Corpus. Figure 3 shows that the model trained on the DeepSeekMath Corpus demonstrates better performance than Proof-Pile-2 at 50B tokens (1 full epoch of Proof-Pile-2), indicating the average quality of DeepSeekMath Corpus is higher.
• Multilingual: The DeepSeekMath Corpus encompasses data in multiple languages, predominantly featuring English and Chinese as the two most represented languages. As shown in Table 1, training on the DeepSeekMath Corpus enhances mathematical reasoning performance in both English and Chinese. In contrast, existing mathematical corpora, which are primarily English-centric, show limited improvement and may even hinder performance in Chinese mathematical reasoning.
• Large-scale: The DeepSeekMath Corpus is several times larger than existing mathematical corpora. As depicted in Figure 3, DeepSeek-LLM 1.3B, when trained on the DeepSeekMath Corpus, shows a steeper learning curve along with more lasting improvements. In contrast, the baseline corpora are much smaller, and have already been repeated multiple rounds during training, with the resulting model performance quickly reaching a plateau.

• 高质量： 我们使用少样本思维链提示，在 8 个数学基准上评估下游性能。如表1所示，在 DeepSeekMath Corpus 上训练的模型表现出明显的性能领先。图3表明，在训练至 50B token（即 Proof-Pile-2 的完整一轮）时，DeepSeekMath Corpus 上训练的模型优于 Proof-Pile-2，这说明 DeepSeekMath Corpus 的平均质量更高。
• 多语言： DeepSeekMath Corpus 包含多种语言的数据，其中英文和中文占比最高。如表1所示，在 DeepSeekMath Corpus 上训练可同时提升英文和中文数学推理性能。相比之下，现有数学语料库主要以英文为中心，对中文数学推理的提升有限，甚至可能损害其性能。
• 大规模： DeepSeekMath Corpus 的规模是现有数学语料库的数倍。如图3所示，DeepSeek-LLM 1.3B 在 DeepSeekMath Corpus 上训练时表现出更陡峭的学习曲线和更持久的提升。相比之下，基线语料库规模小得多，在训练期间已经重复多轮，所得模型的性能很快达到平台期。

2.3 Training and Evaluating DeepSeekMath-Base 7B

In this section, we introduce DeepSeekMath-Base 7B, a base model with strong reasoning abilities, especially in mathematics. Our model is initialized with DeepSeek-Coder-Base-v1.5 7B and trained for 500B tokens. The distribution of the data is as follows: 56% is from the DeepSeekMath Corpus, 4% from AlgebraicStack, 10% from arXiv, 20% is Github code, and the remaining 10% is natural language data from Common Crawl in both English and Chinese. We mainly adopt the training setting specified in Section 2.2.1, except that we set the maximum value of the learning rate to 4.2e-4 and use a batch size of 10M tokens.

在本节中，我们介绍 DeepSeekMath-Base 7B，这是一个具有强大推理能力、尤其擅长数学的基础模型。 我们的模型由 DeepSeek-Coder-Base-v1.5 7B 初始化，并训练 500B token。 数据分布如下：56% 来自 DeepSeekMath Corpus，4% 来自 AlgebraicStack，10% 来自 arXiv，20% 为 Github 代码，其余 10% 是来自 Common Crawl 的中英文自然语言数据。 我们主要采用第 2.2.1 节所述的训练设置，但将最大学习率设为 4.2e-4，并使用 10M token 的批量大小。

We conduct a comprehensive assessment of the mathematical capabilities of DeepSeekMath-Base 7B, focusing on its ability to produce self-contained mathematical solutions without relying on external tools, solve mathematical problems using tools, and conduct formal theorem proving. Beyond mathematics, we also provide a more general profile of the base model, including its performance of natural language understanding, reasoning, and programming skills.

我们全面评估 DeepSeekMath-Base 7B 的数学能力，重点考察它在不依赖外部工具时生成自包含数学解答、使用工具解决数学问题以及进行形式化定理证明的能力。 除数学之外，我们还提供该基础模型更全面的能力画像，包括自然语言理解、推理和编程技能方面的表现。

Mathematical Problem Solving with Step-by-Step Reasoning. We evaluate DeepSeekMath-Base's performance of solving mathematical problems using few-shot chain-of-thought prompting, across eight benchmarks in English and Chinese. These benchmarks encompass quantitative reasoning (e.g., GSM8K, MATH, and CMATH) and multiple-choice problems (e.g., MMLU-STEM and Gaokao-MathQA), covering diverse fields of mathematics from elementary to college-level complexity.

使用逐步推理解决数学问题。 我们使用少样本思维链提示，在八个中英文基准上评估 DeepSeekMath-Base 解决数学问题的性能。 这些基准涵盖定量推理问题（例如 GSM8K、MATH 和 CMATH）与多项选择题（例如 MMLU-STEM 和 Gaokao-MathQA），覆盖从小学到大学难度的多个数学领域。

As shown in Table 2, DeepSeekMath-Base 7B leads in performance across all eight benchmarks among the open-source base models (including the widely-used general model Mistral 7B and the recently released Llemma 34B which underwent math training on Proof-Pile-2). Notably, on the competition-level MATH dataset, DeepSeekMath-Base surpasses existing open-source base models by over 10% absolute, and outperforms Minerva 540B, a closed-source base model 77 times larger which builds on PaLM and is further trained on mathematical texts.

如表2所示，在开源基础模型中，DeepSeekMath-Base 7B 在全部八个基准上均处于领先地位，其中包括广泛使用的通用模型 Mistral 7B，以及近期发布、在 Proof-Pile-2 上进行数学训练的 Llemma 34B。 尤其是在竞赛级 MATH 数据集上，DeepSeekMath-Base 以超过 10 个百分点的绝对优势超过现有开源基础模型，并优于 Minerva 540B；后者是一个规模大 77 倍、基于 PaLM 并进一步在数学文本上训练的闭源基础模型。

表2：DeepSeekMath-Base 7B 与强大基础模型在中英文数学基准上的比较。模型均使用思维链提示进行评估，Minerva 结果引自其原论文。

Model	Size	English Benchmarks					Chinese Benchmarks
		GSM8K	MATH	OCW	SAT	MMLU STEM	CMATH	Gaokao MathCloze	Gaokao MathQA
Closed-Source Base Model
Minerva	7B	16.2%	14.1%	7.7%	-	35.6%	-	-	-
Minerva	62B	52.4%	27.6%	12.0%	-	53.9%	-	-	-
Minerva	540B	58.8%	33.6%	17.6%	-	63.9%	-	-	-
Open-Source Base Model
Mistral	7B	40.3%	14.3%	9.2%	71.9%	51.1%	44.9%	5.1%	23.4%
Llemma	7B	37.4%	18.1%	6.3%	59.4%	43.1%	43.4%	11.9%	23.6%
Llemma	34B	54.0%	25.3%	10.3%	71.9%	52.9%	56.1%	11.9%	26.2%
DeepSeekMath-Base	7B	64.2%	36.2%	15.4%	84.4%	56.5%	71.7%	20.3%	35.3%

Mathematical Problem Solving with Tool Use. We evaluate program-aided mathematical reasoning on GSM8K and MATH using few-shot program-of-thought prompting. Models are prompted to solve each problem by writing a Python program where libraries such as math and sympy can be utilized for intricate computations. The execution result of the program is evaluated as the answer. As shown in Table 3, DeepSeekMath-Base 7B outperforms the prior state-of-the-art Llemma 34B.

使用工具解决数学问题。 我们使用少样本程序思维提示，在 GSM8K 和 MATH 上评估程序辅助数学推理。 模型需要通过编写 Python 程序来解决每个问题，并可使用 math 和 sympy 等库完成复杂计算。 程序的执行结果被作为答案进行评估。 如表3所示，DeepSeekMath-Base 7B 优于此前最先进的 Llemma 34B。

表3：基础模型使用工具解决数学问题以及在 Isabelle 中进行非形式化到形式化定理证明的少样本评估。

Model	Size	Problem Solving w/ Tools		Informal-to-Formal Proving
		GSM8K+Python	MATH+Python	miniF2F-valid	miniF2F-test
Mistral	7B	48.5%	18.2%	18.9%	18.0%
CodeLlama	7B	27.1%	17.2%	16.3%	17.6%
CodeLlama	34B	52.7%	23.5%	18.5%	18.0%
Llemma	7B	41.0%	18.6%	20.6%	22.1%
Llemma	34B	64.6%	26.3%	21.0%	21.3%
DeepSeekMath-Base	7B	66.9%	31.4%	25.8%	24.6%

Formal Mathematics. Formal proof automation is beneficial to ensure the accuracy and reliability of mathematical proofs and enhance efficiency, with increasing attention in recent years. We evaluate DeepSeekMath-Base 7B on the task of informal-to-formal proving which is to generate a formal proof based on an informal statement, a formal counterpart of the statement, and an informal proof. We evaluate on miniF2F, a benchmark for formal Olympiad-level mathematics, and generate a formal proof in Isabelle for each problem with few-shot prompting.

形式数学。 形式化证明自动化有助于确保数学证明的准确性与可靠性并提高效率，近年来受到越来越多关注。 我们在非形式化到形式化证明任务上评估 DeepSeekMath-Base 7B，该任务要求基于非形式化陈述、对应的形式化陈述以及非形式化证明生成形式化证明。 我们在形式化奥林匹克级数学基准 miniF2F 上进行评估，并通过少样本提示为每个问题生成 Isabelle 形式化证明。

We leverage models to generate proof sketches, and execute the off-the-shelf automated prover Sledgehammer to fill in the missing details. As shown in Table 3, DeepSeekMath-Base 7B demonstrates strong performance in proof autoformalization.

我们利用模型生成证明草图，并执行现成的自动证明器 Sledgehammer 来补全缺失细节。 如表3所示，DeepSeekMath-Base 7B 在证明自动形式化方面表现强劲。

表4：自然语言理解、推理和代码基准上的评估。DeepSeek-Coder-Base-v1.5† 是学习率衰减前的检查点，用于训练 DeepSeekMath-Base。MMLU 和 BBH 使用少样本思维链提示；HumanEval 与 MBPP 分别使用零样本和少样本设置。

Model	Size	MMLU	BBH	HumanEval (Pass@1)	MBPP (Pass@1)
Mistral	7B	62.4%	55.7%	28.0%	41.4%
DeepSeek-Coder-Base-v1.5†	7B	42.9%	42.9%	40.2%	52.6%
DeepSeek-Coder-Base-v1.5	7B	49.1%	55.2%	43.2%	60.4%
DeepSeekMath-Base	7B	54.9%	59.5%	40.9%	52.6%

Natural Language Understanding, Reasoning, and Code. We evaluate model performance of natural language understanding on MMLU, reasoning on BBH, and coding capabilities on HumanEval and MBPP. As shown in Table 4, DeepSeekMath-Base 7B exhibits significant enhancements in performance on MMLU and BBH over its precursor, DeepSeek-Coder-Base-v1.5, illustrating the positive impact of math training on language understanding and reasoning. Additionally, by including code tokens for continual training, DeepSeekMath-Base 7B effectively maintains the performance of DeepSeek-Coder-Base-v1.5 on the two coding benchmarks. Overall, DeepSeekMath-Base 7B significantly outperforms the general model Mistral 7B on the three reasoning and coding benchmarks.

自然语言理解、推理与代码。 我们在 MMLU 上评估模型的自然语言理解性能，在 BBH 上评估推理能力，并在 HumanEval 和 MBPP 上评估编码能力。 如表4所示，与前身 DeepSeek-Coder-Base-v1.5 相比，DeepSeekMath-Base 7B 在 MMLU 和 BBH 上的性能显著提高，说明数学训练对语言理解和推理产生了积极影响。 此外，通过在持续训练中加入代码 token，DeepSeekMath-Base 7B 有效保持了 DeepSeek-Coder-Base-v1.5 在两个代码基准上的性能。 总体而言，DeepSeekMath-Base 7B 在三个推理与代码基准上显著优于通用模型 Mistral 7B。

3. Supervised Fine-Tuning

3.1 SFT Data Curation

We construct a mathematical instruction-tuning dataset covering English and Chinese problems from different mathematical fields and of varying complexity levels: problems are paired with solutions in chain-of-thought (CoT), program-of-thought (PoT), and tool-integrated reasoning format. The total number of training examples is 776K.

我们构建了一个数学指令微调数据集，覆盖来自不同数学领域、具有不同复杂度的英文和中文问题：问题分别与思维链（CoT）、程序思维（PoT）和工具集成推理格式的解答配对。 训练样本总数为 776K。

• English mathematical datasets: We annotate GSM8K and MATH problems with tool-integrated solutions, and adopt a subset of MathInstruct along with the training set of Lila-OOD where problems are solved with CoT or PoT. Our English collection covers diverse fields of mathematics, e.g., algebra, probability, number theory, calculus, and geometry.

• 英文数学数据集： 我们为 GSM8K 和 MATH 问题标注工具集成解答，并采用 MathInstruct 的一个子集以及 Lila-OOD 的训练集，其中的问题通过 CoT 或 PoT 求解。我们的英文数据涵盖代数、概率、数论、微积分和几何等多种数学领域。

• Chinese mathematical datasets: We collect Chinese K-12 mathematical problems spanning 76 sub-topics such as linear equations, with solutions annotated in both CoT and tool-integrated reasoning format.

• 中文数学数据集： 我们收集了覆盖线性方程等 76 个子主题的中文 K-12 数学问题，并以 CoT 和工具集成推理两种格式标注解答。

3.2 Training and Evaluating DeepSeekMath-Instruct 7B

In this section, we introduce DeepSeekMath-Instruct 7B which undergoes mathematical instruction tuning based on DeepSeekMath-Base. Training examples are randomly concatenated until reaching a maximum context length of 4K tokens. We train the model for 500 steps with a batch size of 256 and a constant learning rate of 5e-5.

在本节中，我们介绍基于 DeepSeekMath-Base 进行数学指令微调的 DeepSeekMath-Instruct 7B。 训练样本被随机拼接，直至达到 4K token 的最大上下文长度。 我们以 256 的批量大小和恒定的 5e-5 学习率训练模型 500 步。

We evaluate models' mathematical performance both without and with tool use, on 4 quantitative reasoning benchmarks in English and Chinese. We benchmark our model against the leading models of the time:

我们在四个英文和中文定量推理基准上，分别评估模型不使用工具和使用工具时的数学性能。 我们将模型与当时的领先模型进行比较：

• Closed-source models include: (1) the GPT family among which GPT-4 and GPT-4 Code Interpreter are the most capable ones, (2) Gemini Ultra and Pro, (3) Inflection-2, (4) Grok-1, as well as models recently released by Chinese companies including (5) Baichuan-3, (6) the latest GLM-4 from the GLM family. These models are for general purposes, most of which have undergone a series of alignment procedures.

• 闭源模型包括：(1) GPT 系列，其中 GPT-4 和 GPT-4 Code Interpreter 能力最强；(2) Gemini Ultra 和 Pro；(3) Inflection-2；(4) Grok-1；以及中国公司近期发布的模型，包括 (5) Baichuan-3 和 (6) GLM 系列最新的 GLM-4。这些模型均为通用模型，其中大多数经过了一系列对齐流程。

• Open-source models include: general models like (1) DeepSeek-LLM-Chat 67B, (2) Qwen 72B, (3) SeaLLM-v2 7B, and (4) ChatGLM3 6B, as well as models with enhancements in mathematics including (5) InternLM2-Math 20B which builds on InternLM2 and underwent math training followed by instruction tuning, (6) Math-Shepherd-Mistral 7B which applys PPO training to Mistral 7B with a process-supervised reward model, (7) the WizardMath series which improves mathematical reasoning in Mistral 7B and Llama-2 70B using evolve-instruct (i.e., a version of instruction tuning that uses AI-evolved instructions) and PPO training with training problems primarily sourced from GSM8K and MATH, (8) MetaMath 70B which is Llama-2 70B fine-tuned on an augmented version of GSM8K and MATH, (9) ToRA 34B which is CodeLlama 34B fine-tuned to do tool-integrated mathematical reasoning, (10) MAmmoTH 70B which is Llama-2 70B instruction-tuned on MathInstruct.

• 开源模型包括：(1) DeepSeek-LLM-Chat 67B、(2) Qwen 72B、(3) SeaLLM-v2 7B 和 (4) ChatGLM3 6B 等通用模型，以及经过数学能力增强的模型：(5) InternLM2-Math 20B，它基于 InternLM2，先进行数学训练，再进行指令微调；(6) Math-Shepherd-Mistral 7B，它使用过程监督奖励模型对 Mistral 7B 进行 PPO 训练；(7) WizardMath 系列，它使用 evolve-instruct（即采用 AI 演化指令的一种指令微调版本）和 PPO 训练提升 Mistral 7B 与 Llama-2 70B 的数学推理能力，训练问题主要来自 GSM8K 和 MATH；(8) MetaMath 70B，即在增强版 GSM8K 和 MATH 上微调的 Llama-2 70B；(9) ToRA 34B，即经过工具集成数学推理微调的 CodeLlama 34B；(10) MAmmoTH 70B，即在 MathInstruct 上进行指令微调的 Llama-2 70B。

表5：开源与闭源模型在英文和中文基准上使用思维链推理和工具集成推理的性能。灰色分数表示 32 个候选答案的多数投票结果；其余为 Top-1 分数。DeepSeekMath-RL 7B 优于从 7B 到 70B 的所有开源模型以及大多数闭源模型。尽管 DeepSeekMath-RL 7B 仅在 GSM8K 和 MATH 的思维链格式指令微调数据上进一步训练，但它在所有基准上均优于 DeepSeekMath-Instruct 7B。

Model	Size	English Benchmarks		Chinese Benchmarks
		GSM8K	MATH	MGSM-zh	CMATH
Chain-of-Thought Reasoning
Closed-Source Model
Gemini Ultra	-	94.4%	53.2%	-	-
GPT-4	-	92.0%	52.9%	-	86.0%
Inflection-2	-	81.4%	34.8%	-	-
GPT-3.5	-	80.8%	34.1%	-	73.8%
Gemini Pro	-	86.5%	32.6%	-	-
Grok-1	-	62.9%	23.9%	-	-
Baichuan-3	-	88.2%	49.2%	-	-
GLM-4	-	87.6%	47.9%	-	-
Open-Source Model
InternLM2-Math	20B	82.6%	37.7%	-	-
Qwen	72B	78.9%	35.2%	-	-
Math-Shepherd-Mistral	7B	84.1%	33.0%	-	-
WizardMath-v1.1	7B	83.2%	33.0%	-	-
DeepSeek-LLM-Chat	67B	84.1%	32.6%	74.0%	80.3%
MetaMath	70B	82.3%	26.6%	66.4%	70.9%
SeaLLM-v2	7B	78.2%	27.5%	64.8%	-
ChatGLM3	6B	72.3%	25.7%	-	-
WizardMath-v1.0	70B	81.6%	22.7%	64.8%	65.4%
DeepSeekMath-Instruct	7B	82.9%	46.8%	73.2%	84.6%
DeepSeekMath-RL	7B	88.2%	51.7%	79.6%	88.8%
Tool-Integrated Reasoning
Closed-Source Model
GPT-4 Code Interpreter	-	97.0%	69.7%	-	-
Open-Source Model
InternLM2-Math	20B	80.7%	54.3%	-	-
DeepSeek-LLM-Chat	67B	86.7%	51.1%	76.4%	85.4%
ToRA	34B	80.7%	50.8%	41.2%	53.4%
MAmmoTH	70B	76.9%	41.8%	-	-
DeepSeekMath-Instruct	7B	83.7%	57.4%	72.0%	84.3%
DeepSeekMath-RL	7B	86.7%	58.8%	78.4%	87.6%

As shown in Table 5, under the evaluation setting where tool use is disallowed, DeepSeekMath-Instruct 7B demonstrates strong performance of step-by-step reasoning. Notably, on the competition-level MATH dataset, our model surpasses all open-source models and the majority of proprietary models (e.g., Inflection-2 and Gemini Pro) by at least 9% absolute. This is true even for models that are substantially larger (e.g., Qwen 72B) or have been specifically enhanced through math-focused reinforcement learning (e.g., WizardMath-v1.1 7B). While DeepSeekMath-Instruct rivals the Chinese proprietary models GLM-4 and Baichuan-3 on MATH, it still underperforms GPT-4 and Gemini Ultra.

如表5所示，在禁止使用工具的评估设置下，DeepSeekMath-Instruct 7B 展现出很强的逐步推理性能。 值得注意的是，在竞赛级 MATH 数据集上，我们的模型以至少 9 个百分点的绝对优势超过所有开源模型和大多数专有模型（如 Inflection-2 和 Gemini Pro）。 即使面对规模大得多的模型（如 Qwen 72B），或经过数学强化学习专门增强的模型（如 WizardMath-v1.1 7B），这一结论仍然成立。 尽管 DeepSeekMath-Instruct 在 MATH 上可与中国专有模型 GLM-4 和 Baichuan-3 相媲美，但仍落后于 GPT-4 和 Gemini Ultra。

Under the evaluation setting where models are allowed to integrate natural language reasoning and program-based tool use for problem solving, DeepSeekMath-Instruct 7B approaches an accuracy of 60% on MATH, surpassing all existing open-source models. On the other benchmarks, our model is competitive with DeepSeek-LLM-Chat 67B, the prior state-of-the-art that is 10 times larger.

在允许模型结合自然语言推理和程序化工具解决问题的评估设置下，DeepSeekMath-Instruct 7B 在 MATH 上的准确率接近 60%，超过所有现有开源模型。 在其他基准上，我们的模型可与此前最先进、规模大 10 倍的 DeepSeek-LLM-Chat 67B 竞争。

4. Reinforcement Learning

4.1 Group Relative Policy Optimization

Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage. In this section, we introduce our efficient and effective RL algorithm, Group Relative Policy Optimization (GRPO).

强化学习（RL）已被证明能够在监督微调（SFT）阶段之后进一步提升 LLM 的数学推理能力。 在本节中，我们介绍高效且有效的强化学习算法——组相对策略优化（GRPO）。

4.1.1 From PPO to GRPO

Proximal Policy Optimization (PPO) is an actor-critic RL algorithm that is widely used in the RL fine-tuning stage of LLMs. In particular, it optimizes LLMs by maximizing the following surrogate objective:

近端策略优化（PPO）是一种 actor-critic 强化学习算法，广泛用于 LLM 的强化学习微调阶段。 具体而言，它通过最大化以下代理目标来优化 LLM：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{PPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}q\sim P(Q),\\ o\sim {\pi }_{{\theta }_{\mathrm{old}}}(O\mid q)\end{array}}\frac{1}{|o|}\sum _{t=1}^{|o|}min[\frac{{\pi }_{\theta }(o_t\mid q,o_{<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_t\mid q,o_{<t})}{A}_t,\\ & \mathrm{clip}(\frac{{\pi }_{\theta }(o_t\mid q,o_{<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_t\mid q,o_{<t})},1-ϵ,1+ϵ)A_t].\end{array}$$

Here ${\pi }_{\theta }$ and ${\pi }_{{\theta }_{\mathrm{old}}}$ are the current and old policy models, and $q,o$ are questions and outputs sampled from the question dataset and the old policy ${\pi }_{{\theta }_{\mathrm{old}}}$, respectively. $ϵ$ is a clipping-related hyper-parameter introduced in PPO for stabilizing training. $A_t$ is the advantage, which is computed by applying Generalized Advantage Estimation (GAE), based on the rewards $\{r_{\ge t}\}$ and a learned value function $V_{\psi }$. Thus, in PPO, a value function needs to be trained alongside the policy model and to mitigate over-optimization of the reward model, the standard approach is to add a per-token KL penalty from a reference model in the reward at each token, i.e.,

这里，${\pi }_{\theta }$ 和 ${\pi }_{{\theta }_{\mathrm{old}}}$ 分别是当前策略模型和旧策略模型，$q,o$ 分别是从问题数据集和旧策略 ${\pi }_{{\theta }_{\mathrm{old}}}$ 中采样的问题与输出。 $ϵ$ 是 PPO 中用于稳定训练的裁剪相关超参数。 $A_t$ 是优势，它基于奖励 $\{r_{\ge t}\}$ 和学习得到的价值函数 $V_{\psi }$，通过广义优势估计（GAE）计算。 因此，在 PPO 中，需要在训练策略模型的同时训练一个价值函数；为了缓解奖励模型的过度优化，标准做法是在每个 token 的奖励中加入来自参考模型的逐 token KL 惩罚，即：

$$r_t=r_{\varphi }(q,o_{\le t})-\beta \log \frac{{\pi }_{\theta }(o_t\mid q,o_{<t})}{{\pi }_{\mathrm{ref}}(o_t\mid q,o_{<t})}.$$

Here $r_{\varphi }$ is the reward model, ${\pi }_{\mathrm{ref}}$ is the reference model, which is usually the initial SFT model, and $\beta$ is the coefficient of the KL penalty.

这里，$r_{\varphi }$ 是奖励模型，${\pi }_{\mathrm{ref}}$ 是参考模型，通常为初始 SFT 模型，$\beta$ 是 KL 惩罚系数。

图4：PPO 与我们提出的 GRPO 示意图。GRPO 不再使用价值模型，而是根据组内分数估计基线，从而显著减少训练资源。

As the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden. Additionally, during RL training, the value function is treated as a baseline in the calculation of the advantage for variance reduction. While in the LLM context, usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token.

由于 PPO 使用的价值函数通常是另一个与策略模型规模相当的模型，因此会带来巨大的内存和计算负担。 此外，在强化学习训练期间，价值函数在优势计算中被用作降低方差的基线。 而在 LLM 场景中，奖励模型通常只为最后一个 token 分配奖励分数，这可能使训练一个在每个 token 上都准确的价值函数变得复杂。

To address this, as shown in Figure 4, we propose Group Relative Policy Optimization (GRPO), which obviates the need for additional value function approximation as in PPO, and instead uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline. More specifically, for each question $q$, GRPO samples a group of outputs $\{o_1,o_2,\cdots ,o_G\}$ from the old policy ${\pi }_{{\theta }_{\mathrm{old}}}$ and then optimizes the policy model by maximizing the following objective:

为解决这一问题，如图4所示，我们提出组相对策略优化（GRPO）：它不再像 PPO 那样需要额外的价值函数近似，而是将同一问题对应的多个采样输出的平均奖励作为基线。 更具体地说，对于每个问题 $q$，GRPO 从旧策略 ${\pi }_{{\theta }_{\mathrm{old}}}$ 中采样一组输出 $\{o_1,o_2,\cdots ,o_G\}$，然后通过最大化以下目标来优化策略模型：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{GRPO}}(\theta )& ={\mathbb{E}}_{\begin{array}{c}q\sim P(Q),\\ \{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(O\mid q)\end{array}}\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\sum _{t=1}^{|o_i|}\{\\ & min[\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}{\hat{A}}_{i,t},\\ & \mathrm{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})},1-ϵ,1+ϵ){\hat{A}}_{i,t}]\\ & -\beta {\mathbb{D}}_{\mathrm{KL}}[{\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}}]\}.\end{array}$$

Here $ϵ$ and $\beta$ are hyper-parameters, and ${\hat{A}}_{i,t}$ is the advantage calculated based on relative rewards of the outputs inside each group only, which will be detailed in the following subsections. The group relative way that GRPO leverages to calculate the advantages, aligns well with the comparative nature of rewards models, as reward models are typically trained on datasets of comparisons between outputs on the same question. Also note that, instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of ${\hat{A}}_{i,t}$. And different from the KL penalty term used in the PPO reward, we estimate the KL divergence with the following unbiased estimator:

这里，$ϵ$ 和 $\beta$ 是超参数，${\hat{A}}_{i,t}$ 是仅根据各组内部输出的相对奖励计算的优势，后续小节将对此进行详细说明。 GRPO 用于计算优势的组相对方式与奖励模型的比较性质高度契合，因为奖励模型通常在同一问题不同输出之间的比较数据集上训练。 还需注意，GRPO 不在奖励中加入 KL 惩罚，而是将训练策略与参考策略之间的 KL 散度直接加入损失进行正则化，从而避免使 ${\hat{A}}_{i,t}$ 的计算复杂化。 与 PPO 奖励中使用的 KL 惩罚项不同，我们使用以下无偏估计量估计 KL 散度：

$$\begin{array}{rl}{\mathbb{D}}_{\mathrm{KL}}[{\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}}]& =\frac{{\pi }_{\mathrm{ref}}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}\\ & -\log \frac{{\pi }_{\mathrm{ref}}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}-1.\end{array}$$

This estimator is guaranteed to be positive.

该估计量保证为正。

Algorithm 1: Iterative Group Relative Policy Optimization

Input: initial policy model ${\pi }_{{\theta }_{\mathrm{init}}}$; reward models $r_{\varphi }$; task prompts $\mathcal{D}$; hyperparameters $ϵ$, $\beta$, $\mu$

1. policy model ${\pi }_{\theta }\leftarrow {\pi }_{{\theta }_{\mathrm{init}}}$
2. for iteration $=1,\dots ,I$ do
3.   reference model ${\pi }_{\mathrm{ref}}\leftarrow {\pi }_{\theta }$
4.   for step $=1,\dots ,M$ do
5.     Sample a batch ${\mathcal{D}}_b$ from $\mathcal{D}$
6.     Update the old policy model ${\pi }_{{\theta }_{\mathrm{old}}}\leftarrow {\pi }_{\theta }$
7.     Sample $G$ outputs $\{o_i{\}}_{i=1}^G\sim {\pi }_{{\theta }_{\mathrm{old}}}(\cdot \mid q)$ for each question $q\in {\mathcal{D}}_b$
8.     Compute rewards $\{r_i{\}}_{i=1}^G$ for each sampled output $o_i$ by running $r_{\varphi }$
9.     Compute ${\hat{A}}_{i,t}$ for the $t$-th token of $o_i$ through group relative advantage estimation
10.     for GRPO iteration $=1,\dots ,\mu$ do
11.       Update the policy model ${\pi }_{\theta }$ by maximizing the GRPO objective
12.     end for
13.   end for
14.   Update $r_{\varphi }$ through continuous training using a replay mechanism
15. end for
16. return ${\pi }_{\theta }$

4.1.2 Outcome Supervision RL with GRPO

Formally, for each question $q$, a group of outputs $\{o_1,o_2,\cdots ,o_G\}$ are sampled from the old policy model ${\pi }_{{\theta }_{\mathrm{old}}}$. A reward model is then used to score the outputs, yielding $G$ rewards $\mathbf{r}=\{r_1,r_2,\cdots ,r_G\}$ correspondingly. Subsequently, these rewards are normalized by subtracting the group average and dividing by the group standard deviation. Outcome supervision provides the normalized reward at the end of each output $o_i$ and sets the advantages ${\hat{A}}_{i,t}$ of all tokens in the output as the normalized reward, i.e.,

形式化地，对于每个问题 $q$，从旧策略模型 ${\pi }_{{\theta }_{\mathrm{old}}}$ 中采样一组输出 $\{o_1,o_2,\cdots ,o_G\}$。 随后使用奖励模型为这些输出评分，相应得到 $G$ 个奖励 $\mathbf{r}=\{r_1,r_2,\cdots ,r_G\}$。 接下来，通过减去组内平均值并除以组内标准差，对这些奖励进行归一化。 结果监督在每个输出 $o_i$ 的末尾提供归一化奖励，并将该输出中所有 token 的优势 ${\hat{A}}_{i,t}$ 设为归一化奖励，即：

$${\hat{A}}_{i,t}={\tilde{r}}_i=\frac{r_i-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}.$$

It then optimizes the policy by maximizing the GRPO objective.

随后通过最大化 GRPO 目标来优化策略。

4.1.3 Process Supervision RL with GRPO

Outcome supervision only provides a reward at the end of each output, which may not be sufficient and efficient to supervise the policy in complex mathematical tasks. We also explore process supervision, which provides a reward at the end of each reasoning step. Formally, given the question $q$ and $G$ sampled outputs $\{o_1,o_2,\cdots ,o_G\}$, a process reward model is used to score each step of the outputs, yielding corresponding rewards:

结果监督只在每个输出末尾提供奖励，这对于复杂数学任务中的策略监督可能不够充分且效率不高。 我们还探索了过程监督，它在每个推理步骤末尾提供奖励。 形式化地，给定问题 $q$ 和 $G$ 个采样输出 $\{o_1,o_2,\cdots ,o_G\}$，使用过程奖励模型为输出的每个步骤评分，得到相应奖励：

$$\begin{array}{rl}\mathbf{R}=\{& \{r_1^{\mathrm{index}(1)},\dots ,r_1^{\mathrm{index}(K_1)}\},\dots ,\\ & \{r_G^{\mathrm{index}(1)},\dots ,r_G^{\mathrm{index}(K_G)}\}\}.\end{array}$$

Here $\mathrm{index}(j)$ is the end token index of the $j$-th step, and $K_i$ is the total number of steps in the $i$-th output. We also normalize these rewards with the average and the standard deviation, i.e.,

这里，$\mathrm{index}(j)$ 是第 $j$ 个步骤末尾 token 的索引，$K_i$ 是第 $i$ 个输出中的步骤总数。 我们同样使用平均值和标准差对这些奖励进行归一化，即：

$${\tilde{r}}_i^{\mathrm{index}(j)}=\frac{r_i^{\mathrm{index}(j)}-\mathrm{mean}(\mathbf{R})}{\mathrm{std}(\mathbf{R})}.$$

Subsequently, the process supervision calculates the advantage of each token as the sum of the normalized rewards from the following steps, i.e.,

随后，过程监督将每个 token 的优势计算为后续步骤归一化奖励之和，即：

$${\hat{A}}_{i,t}=\sum _{\mathrm{index}(j)\ge t}{\tilde{r}}_i^{\mathrm{index}(j)}.$$

It then optimizes the policy by maximizing the GRPO objective.

随后通过最大化 GRPO 目标来优化策略。

4.1.4 Iterative RL with GRPO

As the reinforcement learning training process progresses, the old reward model may not be sufficient to supervise the current policy model. Therefore, we also explore the iterative RL with GRPO. As shown in Algorithm 1, in iterative GRPO, we generate new training sets for the reward model based on the sampling results from the policy model and continually train the old reward model using a replay mechanism that incorporates 10% of historical data. Then, we set the reference model as the policy model, and continually train the policy model with the new reward model.

随着强化学习训练过程推进，旧奖励模型可能不足以监督当前策略模型。 因此，我们还探索了使用 GRPO 的迭代强化学习。 如算法1所示，在迭代式 GRPO 中，我们根据策略模型的采样结果为奖励模型生成新训练集，并使用纳入 10% 历史数据的回放机制持续训练旧奖励模型。 随后，我们将参考模型设为策略模型，并使用新的奖励模型持续训练策略模型。

4.2 Training and Evaluating DeepSeekMath-RL

We conduct RL based on DeepSeekMath-Instruct 7B. The training data of RL are chain-of-thought-format questions related to GSM8K and MATH from the SFT data, which consists of around 144K questions. We exclude other SFT questions to investigate the impact of RL on benchmarks that lack data throughout the RL phase. We construct the training set of reward models following Wang et al. We train our initial reward model based on the DeepSeekMath-Base 7B with a learning rate of 2e-5.

我们基于 DeepSeekMath-Instruct 7B 进行强化学习。 强化学习训练数据来自 SFT 数据中与 GSM8K 和 MATH 相关的思维链格式问题，共约 144K 个问题。 我们排除其他 SFT 问题，以研究强化学习对在整个强化学习阶段均缺少数据的基准所产生的影响。 我们按照 Wang 等人的方法构建奖励模型训练集。 我们基于 DeepSeekMath-Base 7B 训练初始奖励模型，学习率为 2e-5。

For GRPO, we set the learning rate of the policy model as 1e-6. The KL coefficient is 0.04. For each question, we sample $64$ outputs. The max length is set to 1024, and the training batch size is 1024. The policy model only has a single update following each exploration stage.

对于 GRPO，我们将策略模型的学习率设为 1e-6。 KL 系数为 0.04。 对于每个问题，我们采样 $64$ 个输出。 最大长度设为 1024，训练批量大小也为 1024。 策略模型在每个探索阶段之后只更新一次。

We evaluate DeepSeekMath-RL 7B on benchmarks following DeepSeekMath-Instruct 7B. For DeepSeekMath-RL 7B, GSM8K and MATH with chain-of-thought reasoning can be regarded as in-domain tasks and all the other benchmarks can be regarded as out-of-domain tasks. Table 5 demonstrates the performance of open- and closed-source models with both chain-of-thought and tool-integrated reasoning on English and Chinese benchmarks. We find that:

我们沿用 DeepSeekMath-Instruct 7B 的方式，在各基准上评估 DeepSeekMath-RL 7B。 对于 DeepSeekMath-RL 7B，采用思维链推理的 GSM8K 和 MATH 可视为域内任务，其他所有基准均可视为域外任务。 表5展示了开源与闭源模型在英文和中文基准上使用思维链推理和工具集成推理的性能。 我们发现：

1. DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respectively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source models in the 7B to 70B range, as well as the majority of closed-source models.

1. DeepSeekMath-RL 7B 使用思维链推理时，在 GSM8K 和 MATH 上分别达到 88.2% 和 51.7% 的准确率。这一性能超过 7B 至 70B 范围内的所有开源模型以及大多数闭源模型。

2. Crucially, DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Despite the constrained scope of its training data, it outperforms DeepSeekMath-Instruct 7B across all evaluation metrics, showcasing the effectiveness of reinforcement learning.

2. 关键在于，DeepSeekMath-RL 7B 以 DeepSeekMath-Instruct 7B 为起点，仅在 GSM8K 和 MATH 的思维链格式指令微调数据上进行训练。尽管训练数据范围有限，它仍在所有评估指标上优于 DeepSeekMath-Instruct 7B，展现了强化学习的有效性。

5. Discussion

In this section, we will share our findings in pre-training and RL experiments.

在本节中，我们将分享预训练与强化学习实验中的发现。

5.1 Lessons Learnt in Pre-Training

We first share our experience in pre-training. Unless otherwise specified, we will adhere to the training settings outlined in Section 2.2. It is worth noting that, when referring to the DeepSeekMath Corpus in this section, we use an 89B-token dataset from the second iteration of the data collection process.

我们首先分享预训练方面的经验。 除非另有说明，我们将沿用第 2.2 节所述的训练设置。 需要说明的是，本节提到 DeepSeekMath Corpus 时，使用的是数据收集流程第二轮迭代得到的 89B-token 数据集。

5.1.1 Code Training Benefits Mathematical Reasoning

A popular yet unverified hypothesis suggests that code training improves reasoning. We attempt to offer a partial response to this, particularly within the mathematical domain: code training improves models' ability to do mathematical reasoning both with and without tool use. To study how code training affects mathematical reasoning, we experimented with the following two-stage training and one-stage training settings:

一种流行但尚未得到验证的假设认为，代码训练能够改善推理。 我们试图对此给出部分回答，尤其是在数学领域：代码训练能够提升模型在使用和不使用工具两种情况下进行数学推理的能力。 为了研究代码训练如何影响数学推理，我们采用了以下两阶段训练和单阶段训练设置：

Two-Stage Training

两阶段训练

• Code Training for 400B Tokens $\to$ Math Training for 150B Tokens: We train DeepSeek-LLM 1.3B for 400B code tokens followed by 150B math tokens.

• 训练 400B 代码 token $\to$ 训练 150B 数学 token： 我们先使用 400B 代码 token 训练 DeepSeek-LLM 1.3B，再使用 150B 数学 token 训练。

• General Training for 400B Tokens $\to$ Math Training for 150B Tokens: As a control experiment, we also experiment with general tokens (sampled from a large-scale general corpus created by DeepSeek-AI) instead of code tokens in the first stage of training, in an attempt to investigate the advantages of code tokens over general tokens in improving mathematical reasoning.

• 训练 400B 通用 token $\to$ 训练 150B 数学 token： 作为对照实验，我们还在第一阶段使用通用 token（从 DeepSeek-AI 创建的大规模通用语料库中采样）代替代码 token，以研究代码 token 相较于通用 token 在提升数学推理方面的优势。

One-Stage Training

单阶段训练

• Math Training for 150B Tokens: We train DeepSeek-LLM 1.3B for 150B math tokens.

• 训练 150B 数学 token： 我们使用 150B 数学 token 训练 DeepSeek-LLM 1.3B。

• Training on a mixture of 400B Code Tokens and 150B Math Tokens: Math training following code training degrades coding performance. We investigate whether code tokens, when mixed with math tokens for one-stage training, would still improve mathematical reasoning and also alleviate the problem of catastrophic forgetting.

• 在 400B 代码 token 与 150B 数学 token 的混合数据上训练： 在代码训练之后进行数学训练会降低编码性能。我们研究在单阶段训练中将代码 token 与数学 token 混合，是否仍能改善数学推理，并缓解灾难性遗忘问题。

表6：不同训练设置下代码如何影响数学推理。我们使用 DeepSeek-LLM 1.3B 进行实验，并分别通过少样本思维链提示和少样本程序思维提示，评估其不使用工具和使用工具时的数学推理性能。

Training Setting	Training Tokens			w/o Tool Use			w/ Tool Use
	General	Code	Math	GSM8K	MATH	CMATH	GSM8K + Python	MATH + Python
No Continual Training	-	-	-	2.9%	3.0%	12.3%	2.7%	2.3%
Two-Stage Training
Stage 1: General Training	400B	-	-	2.9%	3.2%	14.8%	3.3%	2.3%
Stage 2: Math Training	-	-	150B	19.1%	14.4%	37.2%	14.3%	6.7%
Stage 1: Code Training	-	400B	-	5.9%	3.6%	19.9%	12.4%	10.0%
Stage 2: Math Training	-	-	150B	21.9%	15.3%	39.7%	17.4%	9.4%
One-Stage Training
Math Training	-	-	150B	20.5%	13.1%	37.6%	11.4%	6.5%
Code & Math Mixed Training	-	400B	150B	17.6%	12.1%	36.3%	19.7%	13.5%

表7：不同代码与数学训练设置对模型语言理解、推理和编码性能的影响。我们使用 DeepSeek-LLM 1.3B 进行实验；MMLU 和 BBH 使用少样本思维链提示，HumanEval 和 MBPP 分别采用零样本与少样本评估。

Training Setting	Training Tokens			MMLU	BBH	HumanEval (Pass@1)	MBPP (Pass@1)
	General	Code	Math
No Continual Training	-	-	-	24.5%	28.1%	12.2%	13.0%
Two-Stage Training
Stage 1: General Training	400B	-	-	25.9%	27.7%	15.2%	13.6%
Stage 2: Math Training	-	-	150B	33.1%	32.7%	12.8%	13.2%
Stage 1: Code Training	-	400B	-	25.0%	31.5%	25.0%	40.0%
Stage 2: Math Training	-	-	150B	36.2%	35.3%	12.2%	17.0%
One-Stage Training
Math Training	-	-	150B	32.3%	32.5%	11.6%	13.2%
Code & Math Mixed Training	-	400B	150B	33.5%	35.6%	29.3%	39.4%

Results. Table 6 and Table 7 demonstrate the downstream performance under different training settings.

结果。 表6和表7展示了不同训练设置下的下游性能。

Code training benefits program-aided mathematical reasoning, both under the two-stage training and one-stage training settings. As shown in Table 6, under the two-stage training setting, code training alone already significantly enhances the ability to solve GSM8K and MATH problems using Python. Math training in the second stage yields further improvements. Interestingly, under the one-stage training setting, mixing code tokens and math tokens effectively mitigates the issue of catastrophic forgetting that arises from two-stage training, and also synergizes coding (Table 7) and program-aided mathematical reasoning (Table 6).

在两阶段训练和单阶段训练设置下，代码训练都有利于程序辅助数学推理。 如表6所示，在两阶段训练设置下，仅进行代码训练就已显著增强使用 Python 解决 GSM8K 和 MATH 问题的能力。 第二阶段的数学训练带来了进一步提升。 有趣的是，在单阶段训练设置下，混合代码 token 与数学 token 有效缓解了两阶段训练产生的灾难性遗忘问题，并使编码能力（表7）与程序辅助数学推理（表6）产生协同效应。

Code training also improves mathematical reasoning without tool use. Under the two-stage training setting, the initial stage of code training already results in moderate enhancements. It also boosts the efficiency of the subsequent math training, eventually leading to the best performance. However, combining code tokens and math tokens for one-stage training compromises mathematical reasoning without tool use. One conjecture is that DeepSeek-LLM 1.3B, due to its limited scale, lacks the capacity to fully assimilate both code and mathematical data simultaneously.

代码训练也能改善不使用工具时的数学推理。 在两阶段训练设置下，初始代码训练阶段已经带来了适度提升。 它还提高了后续数学训练的效率，最终取得最佳性能。 然而，在单阶段训练中结合代码 token 和数学 token 会损害不使用工具时的数学推理。 一种推测是，DeepSeek-LLM 1.3B 受限于模型规模，没有能力同时充分吸收代码数据与数学数据。

5.1.2 ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning

ArXiv papers are commonly included as a component of math pre-training data. However, detailed analysis regarding their impact on mathematical reasoning has not been extensively conducted. Perhaps counter-intuitively, according to our experiments, arXiv papers seem ineffective in improving mathematical reasoning. We experiment with models of different sizes, including DeepSeek-LLM 1.3B and DeepSeek-Coder-Base-v1.5 7B, using arXiv corpora that underwent varied processing pipelines:

arXiv 论文通常被纳入数学预训练数据。 然而，关于它们对数学推理影响的详细分析尚未得到广泛开展。 根据我们的实验，一个可能违反直觉的结果是：arXiv 论文似乎无法有效改善数学推理。 我们使用经过不同处理流水线的 arXiv 语料库，对 DeepSeek-LLM 1.3B 和 DeepSeek-Coder-Base-v1.5 7B 等不同规模的模型进行实验：

• MathPile: an 8.9B-token corpus developed with cleaning and filtering heuristic rules, over 85% of which are scientific arXiv papers.

• MathPile： 一个通过启发式清洗与过滤规则构建的 8.9B-token 语料库，其中超过 85% 为 arXiv 科学论文。

• ArXiv-RedPajama: the entirety of arXiv LaTeX files with preambles, comments, macros, and bibliographies removed, totaling 28.0B tokens.

• ArXiv-RedPajama： 删除导言区、注释、宏和参考文献后的全部 arXiv LaTeX 文件，共计 28.0B token。

In our experiments, we separately train DeepSeek-LLM 1.3B for 150B tokens and DeepSeek-Coder-Base-v1.5 7B for 40B tokens on each arXiv corpus. It seems that arXiv papers are ineffective in improving mathematical reasoning. When trained on a arXiv-only corpus, both models display no notable improvements or even deterioration across various mathematical benchmarks of different complexities employed in this study. These benchmarks include quantitative reasoning datasets like GSM8K and MATH (Table 8), multiple-choice challenges like MMLU-STEM (Table 8), and formal mathematics like miniF2F (Table 9).

在实验中，我们分别在每个 arXiv 语料库上使用 150B token 训练 DeepSeek-LLM 1.3B，并使用 40B token 训练 DeepSeek-Coder-Base-v1.5 7B。 arXiv 论文似乎无法有效改善数学推理。 仅在 arXiv 语料库上训练时，两个模型在本研究采用的多个不同复杂度数学基准上均未呈现显著提升，甚至出现性能下降。 这些基准包括 GSM8K 和 MATH 等定量推理数据集（表8）、MMLU-STEM 等多项选择挑战（表8），以及 miniF2F 等形式化数学任务（表9）。

表8：不同 arXiv 数据集上的数学训练效果。模型性能使用少样本思维链提示进行评估。

Model	Size	ArXiv Corpus	English Benchmarks					Chinese Benchmarks
			GSM8K	MATH	OCW	SAT	MMLU STEM	CMATH	Gaokao MathCloze	Gaokao MathQA
DeepSeek-LLM	1.3B	No Math Training	2.9%	3.0%	2.9%	15.6%	19.5%	12.3%	0.8%	17.9%
		MathPile	2.7%	3.3%	2.2%	12.5%	15.7%	1.2%	0.0%	2.8%
		ArXiv-RedPajama	3.3%	3.4%	4.0%	9.4%	9.0%	7.4%	0.8%	2.3%
DeepSeek-Coder-Base-v1.5	7B	No Math Training	29.0%	12.5%	6.6%	40.6%	38.1%	45.9%	5.9%	21.1%
		MathPile	23.6%	11.5%	7.0%	46.9%	35.8%	37.9%	4.2%	25.6%
		ArXiv-RedPajama	28.1%	11.1%	7.7%	50.0%	35.2%	42.6%	7.6%	24.8%

表9：不同 arXiv 语料库上的数学训练效果，基础模型为 DeepSeek-Coder-Base-v1.5 7B。我们评估 Isabelle 中的非形式化到形式化证明。

ArXiv Corpus	miniF2F-valid	miniF2F-test
No Math Training	20.1%	21.7%
MathPile	16.8%	16.4%
ArXiv-RedPajama	14.8%	11.9%

However, this conclusion has its limitations and should be taken with a grain of salt. We have not yet studied:

然而，这一结论存在局限，应当谨慎看待。 我们尚未研究：

• The impact of arXiv tokens on specific math-related tasks not included in this research, such as informalization of theorems which is to convert formal statements or proofs to their informal versions.

• arXiv token 对本研究未涵盖的特定数学相关任务的影响，例如将形式化陈述或证明转换为非形式化版本的定理非形式化。

• The effect of arXiv tokens when combined with other types of data.

• arXiv token 与其他类型数据结合时的效果。

• Whether the benefits of arXiv papers would manifest themselves at a larger model scale.

• arXiv 论文的益处是否会在更大的模型规模下显现。

Thus, further exploration is required, which we leave for future studies.

因此仍需进一步探索，我们将其留待未来研究。

5.2 Insights of Reinforcement Learning

5.2.1 Towards to a Unified Paradigm

In this section, we provide a unified paradigm to analyze different training methods, such as SFT, RFT, DPO, PPO, GRPO, and further conduct experiments to explore the factors of the unified paradigm. Generally, the gradient with respect to the parameter $\theta$ of a training method can be written as:

在本节中，我们提供一个统一范式来分析 SFT、RFT、DPO、PPO 和 GRPO 等不同训练方法，并进一步开展实验来探索该统一范式中的影响因素。 一般而言，一种训练方法关于参数 $\theta$ 的梯度可写为：

$$\begin{array}{rl}{\mathrm{\nabla }}_{\theta }{\mathcal{J}}_{\mathcal{A}}(\theta )& ={\mathbb{E}}_{(q,o)\sim \mathcal{D}}[\frac{1}{|o|}\sum _{t=1}^{|o|}G{C}_{\mathcal{A}}(q,o,t,{\pi }_{rf}){\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(o_t\mid q,o_{<t})].\end{array}$$

There exist three key components:

1. Data Source $\mathcal{D}$, which determines the training data;
2. Reward Function ${\pi }_{rf}$, which is the source of the training reward signal;
3. Algorithm $\mathcal{A}$: which processes the training data and the reward signal to the gradient coefficient $GC$ that determines the magnitude of the penalty or reinforcement for the data.

We analyze several representative methods based on such a unified paradigm:

其中包含三个关键组成部分：

1. 数据源 $\mathcal{D}$，它决定训练数据；
2. 奖励函数 ${\pi }_{rf}$，它是训练奖励信号的来源；
3. 算法 $\mathcal{A}$：它处理训练数据和奖励信号，得到梯度系数 $GC$，该系数决定对数据施加惩罚或强化的幅度。

我们基于这一统一范式分析若干代表性方法：

表10：不同方法的数据源与梯度系数。$P_{sft}$ 表示监督微调数据集的数据分布；${\pi }_{{\theta }_{sft}}$ 与 ${\pi }_{\theta }$ 分别表示监督微调模型和在线训练过程中的实时策略模型。

Methods	Data Source	Reward Function	Gradient Coefficient
SFT	$q,o\sim P_{sft}(Q,O)$	-	1
RFT	$q\sim P_{sft}(Q),o\sim {\pi }_{sft}(O\mid q)$	Rule	RFT coefficient
DPO	$q\sim P_{sft}(Q),o^{+},o^{-}\sim {\pi }_{sft}(O\mid q)$	Rule	DPO coefficient
Online RFT	$q\sim P_{sft}(Q),o\sim {\pi }_{\theta }(O\mid q)$	Rule	RFT coefficient
PPO	$q\sim P_{sft}(Q),o\sim {\pi }_{\theta }(O\mid q)$	Model	PPO coefficient
GRPO	$q\sim P_{sft}(Q),\{o_i{\}}_{i=1}^G\sim {\pi }_{\theta }(O\mid q)$	Model	GRPO coefficient

• Supervised Fine-tuning (SFT): SFT fine-tunes pretrained model on human selected SFT data.

• 监督微调（SFT）： SFT 在人工选择的 SFT 数据上微调预训练模型。

• Rejection Sampling Fine-tuning (RFT): RFT further fine-tunes the SFT model on the filtered outputs sampled from the SFT model based on SFT questions. RFT filters the outputs based on the correctness of their answers.

• 拒绝采样微调（RFT）： RFT 基于 SFT 问题从 SFT 模型采样输出，并在筛选后的输出上进一步微调 SFT 模型。RFT 根据答案的正确性筛选输出。

• Direct Preference Optimization (DPO): DPO further refines the SFT model by fine-tuning it on augmented outputs sampled from the SFT model, using pair-wise DPO loss.

• 直接偏好优化（DPO）： DPO 使用成对 DPO 损失，在从 SFT 模型采样得到的增强输出上进行微调，从而进一步改进 SFT 模型。

• Online Rejection Sampling Fine-tuning (Online RFT): Different from RFT, Online RFT initiates the policy model using the SFT model and refines it by fine-tuning with the augmented outputs sampled from the real-time policy model.

• 在线拒绝采样微调（Online RFT）： 与 RFT 不同，Online RFT 使用 SFT 模型初始化策略模型，并利用从实时策略模型采样得到的增强输出进行微调来改进该模型。

• PPO/GRPO: PPO/GRPO initializes the policy model using the SFT model and reinforces it with the outputs sampled from the real-time policy model.

• PPO/GRPO： PPO/GRPO 使用 SFT 模型初始化策略模型，并利用从实时策略模型采样的输出对其进行强化。

We summarize the components of these methods in Table 10. Please refer to the appendix for a more detailed derivation process.

我们在 表10 中总结了这些方法的组成部分。 更详细的推导过程见附录。

图5：DeepSeekMath-Instruct 1.3B 模型使用不同方法进一步训练后在两个基准上的性能。

图6：在两个基准上使用 DeepSeekMath-Instruct 7B 进行迭代强化学习的性能。

Observation about Data Source. We divide the data source into two categories, online sampling, and offline sampling. Online sampling denotes that the training data is from the exploration results of the real-time training policy model, while offline sampling denotes that the training data is from the sampling results of the initial SFT model. RFT and DPO follow the offline style, while Online RFT and GRPO follow the online style.

关于数据源的观察。 我们将数据源分为在线采样和离线采样两类。 在线采样表示训练数据来自实时训练策略模型的探索结果，而离线采样表示训练数据来自初始 SFT 模型的采样结果。 RFT 和 DPO 采用离线方式，而 Online RFT 和 GRPO 采用在线方式。

As shown in Figure 5, we find that the Online RFT significantly outperforms RFT on two benchmarks. Specifically, Online RFT is comparable to RFT in the early stage of training but gains an absolute advantage in the later stage, demonstrating the superiority of online training. This is intuitive, as in the initial stage, the actor and the SFT model exhibit close resemblance, with the sampled data revealing only minor differences. In the later stage, however, the data sampled from the actor will exhibit more significant differences, and real-time data sampling will offer greater advantages.

如 图5 所示，我们发现 Online RFT 在两个基准上都显著优于 RFT。 具体而言，Online RFT 在训练早期与 RFT 表现相当，但在后期取得绝对优势，体现了在线训练的优越性。 这很直观，因为在初始阶段，actor 与 SFT 模型非常相似，采样数据只表现出细微差异。 然而到了后期，从 actor 采样的数据会呈现更明显的差异，实时数据采样也会展现出更大优势。

Observation about Gradient Coefficient. The algorithm processes the reward signal to the gradient coefficient to update the model parameter. We divide the reward function as Rule and Model in our experiments. Rule refers to judging the quality of a response based on the correctness of the answer, and Model denotes that we train a reward model to score each response. The training data of the reward model is based on the rule judgment. The corresponding coefficient equations highlight a key difference between GRPO and Online RFT: GRPO uniquely adjusts its gradient coefficient based on the reward value provided by the reward model. This allows for differential reinforcement and penalization of responses according to their varying magnitudes. In contrast, Online RFT lacks this feature; it does not penalize incorrect responses and uniformly reinforces all responses with correct answers at the same level of intensity.

关于梯度系数的观察。 算法将奖励信号处理为梯度系数，以更新模型参数。 在实验中，我们将奖励函数分为 Rule 和 Model。 Rule 是指根据答案正确性判断响应质量，而 Model 表示我们训练奖励模型为每个响应评分。 奖励模型的训练数据基于规则判断得到。 相应的系数方程凸显了 GRPO 与 Online RFT 的一个关键区别：GRPO 会根据奖励模型给出的奖励值调整梯度系数。 这使其能够依据不同幅度对响应实施差异化强化和惩罚。 相比之下，Online RFT 不具备这一特性；它不会惩罚错误响应，并以相同强度统一强化所有答案正确的响应。

As demonstrated in Figure 5, GRPO surpasses online RFT, thereby highlighting the efficiency of altering positive and negative gradient coefficients. In addition, GRPO+PS shows superior performance compared to GRPO+OS, indicating the benefits of using fine-grained, step-aware gradient coefficients. Furthermore, we explore the iterative RL, in our experiments, we conduct two rounds of iteration. As shown in Figure 6, we notice that the iterative RL significantly improves the performance, especially at the first iteration.

如 图5 所示，GRPO 优于 Online RFT，说明调整正负梯度系数是有效的。 此外，GRPO+PS 的表现优于 GRPO+OS，体现了采用细粒度、步骤感知梯度系数的益处。 此外，我们探索了迭代强化学习，并在实验中进行了两轮迭代。 如 图6 所示，我们观察到迭代强化学习显著提升了性能，尤其是在第一次迭代中。

图7：SFT 与 RL DeepSeekMath 7B 在 GSM8K 和 MATH 上的 Maj@K 与 Pass@K（温度为 0.7）。强化学习提高了 Maj@K，但没有提高 Pass@K。

5.2.2 Why RL Works?

In this paper, we conduct reinforcement learning based on a subset of instruction tuning data, and it achieves significant performance enhancement upon the instruction tuning model. To further explain why reinforcement learning works. We evaluate the Pass@K and Maj@K accuracy of the Instruct and RL models on two benchmarks. As shown in Figure 7, RL enhances Maj@K’s performance but not Pass@K. These findings indicate that RL enhances the model's overall performance by rendering the output distribution more robust, in other words, it seems that the improvement is attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities. Similarly, previous work identified a misalignment problem in reasoning tasks within the SFT model, showing that the reasoning performance of SFT models can be improved through a series of preference alignment strategies.

在本文中，我们基于一部分指令微调数据开展强化学习，并在指令微调模型之上取得了显著性能提升。 为了进一步解释强化学习为何有效。 我们在两个基准上评估 Instruct 模型与 RL 模型的 Pass@K 和 Maj@K 准确率。 如 图7 所示，强化学习提高了 Maj@K 的性能，却没有提高 Pass@K。 这些发现表明，强化学习通过使输出分布更加稳健来提高模型整体性能；换言之，这种提升似乎源于将 TopK 中的正确响应推到更靠前的位置，而非基础能力的增强。 类似地，先前工作发现 SFT 模型在推理任务中存在错位问题，并表明可以通过一系列偏好对齐策略提高 SFT 模型的推理性能。

5.2.3 How to Achieve More Effective RL?

We demonstrate RL works pretty well in mathematical reasoning tasks. We also provide a unified paradigm to understand different representative training methods. Within this paradigm, all methods are conceptualized as either direct or simplified RL techniques. As summarized in the objective above, there exist three key components: Data Source, Algorithm, and Reward Function. We provide some potential future directions about the three components.

我们证明强化学习在数学推理任务中表现良好。 我们还提供了一个统一范式，用以理解不同的代表性训练方法。 在这一范式中，所有方法都被视为直接或简化的强化学习技术。 如上面的目标函数所总结，其中包含数据源、算法和奖励函数三个关键组成部分。 我们围绕这三个组成部分提出一些潜在的未来方向。

Data Source. Data source is the raw material of all training methods. In the context of RL, we specifically refer to the data source as the unlabeled questions with the outputs sampled from the policy model. In this paper, we only use the questions from the instruction tuning stage and a naive nucleus sampling to sample outputs. We think this is a potential reason that our RL pipeline only improves the Maj@K performance. In the future, we will explore our RL pipeline on out-of-distribution question prompts, in conjunction with advanced sampling (decoding) strategies, like those based on tree-search methods. Also, the efficient inference techniques, which determines the exploration efficiency of policy models, also play an exceedingly important role.

数据源。 数据源是所有训练方法的原材料。 在强化学习语境下，我们特指由未标注问题和策略模型采样输出组成的数据源。 在本文中，我们只使用指令微调阶段的问题，并采用朴素的核采样来生成输出。 我们认为，这可能是强化学习流水线只提高 Maj@K 性能的原因之一。 未来，我们将结合基于树搜索等方法的高级采样（解码）策略，在分布外问题提示上探索我们的强化学习流水线。 此外，决定策略模型探索效率的高效推理技术同样发挥着极其重要的作用。

Algorithms. Algorithms process the data and reward signal to the gradient coefficient to update the model parameter. Based on the objective above, to some extent, all methods now fully TRUST the signal of the reward function to increase or decrease the conditional probability of a certain token. However, it is impossible to ensure the reward signal is always reliable, especially in extremely complex tasks. For example, even the PRM800K datasets, which have been carefully annotated by well-trained annotators, still contain approximately 20% of incorrectly annotations. To this end, we will explore the reinforcement learning algorithm that is robust against noisy reward signals. We believe such WEAK-TO-STRONG alignment methods will bring a fundamental change to the learning algorithms.

算法。 算法将数据和奖励信号处理为梯度系数，以更新模型参数。 根据上面的目标函数，在某种程度上，现有所有方法都完全信任奖励函数的信号，并据此增大或减小某个 token 的条件概率。 然而，奖励信号不可能始终可靠，尤其是在极其复杂的任务中。 例如，即使 PRM800K 数据集经过训练有素的标注者仔细标注，仍包含约 20% 的错误标注。 为此，我们将探索对噪声奖励信号具有鲁棒性的强化学习算法。 我们相信，这类由弱到强的对齐方法将为学习算法带来根本性变化。

Reward Function. Reward function is the source of the training signal. In RL, the reward function is usually the neural reward model. We think there exist three important directions for reward models:

奖励函数。 奖励函数是训练信号的来源。 在强化学习中，奖励函数通常是神经奖励模型。 我们认为奖励模型存在三个重要方向：

1. How to enhance the generalization ability of the reward model. The reward model must be effectively generalized to handle out-of-distribution questions and advanced decoding outputs; otherwise, reinforcement learning may merely stabilize the distribution of LLMs rather than improve their fundamental capabilities;
2. How to reflect the uncertainty of reward model. The uncertainty could potentially act as a linking bridge between the weak reward model and the weak-to-strong learning algorithms;
3. How to efficiently build high-quality process reward models that can provide fine-grained training signals for the reasoning process.

1. 如何增强奖励模型的泛化能力。 奖励模型必须能够有效泛化，以处理分布外问题和高级解码输出；否则，强化学习可能只会稳定 LLM 的分布，而不能提升其基础能力；
2. 如何反映奖励模型的不确定性。 不确定性可能成为弱奖励模型与由弱到强学习算法之间的连接桥梁；
3. 如何高效构建高质量的过程奖励模型，使其能够为推理过程提供细粒度训练信号。

6. Conclusion, Limitation, and Future Work

We present DeepSeekMath, which outperforms all open-source models on the competition-level MATH benchmark and approaches the performance of closed models. DeepSeekMath is initialized with DeepSeek-Coder-v1.5 7B and undergoes continual training for 500B tokens, with a significant component of the training data being 120B math tokens sourced from Common Crawl. Our extensive ablation study shows web pages offer significant potential for high-quality mathematical data, while arXiv may not as beneficial as we expected. We introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), which can notably improve mathematical reasoning capabilities with less memory consumption. The experiment results show that GRPO is effective even if DeepSeekMath-Instruct 7B has reached a high score on benchmarks. We also provide a unified paradigm to understand a series of methods and summarize several potential directions for more effective reinforcement learning.

我们提出 DeepSeekMath，它在竞赛级 MATH 基准上优于所有开源模型，并接近闭源模型的性能。 DeepSeekMath 以 DeepSeek-Coder-v1.5 7B 初始化，并继续训练 500B token，其中训练数据的重要组成部分是来自 Common Crawl 的 120B 数学 token。 我们广泛的消融研究表明，网页蕴含高质量数学数据的巨大潜力，而 arXiv 可能不像我们预期的那样有益。 我们提出组相对策略优化（GRPO），它是近端策略优化（PPO）的一种变体，能够以更少的内存消耗显著提升数学推理能力。 实验结果表明，即使 DeepSeekMath-Instruct 7B 已在基准上取得较高分数，GRPO 仍然有效。 我们还提供了一个统一范式来理解一系列方法，并总结了实现更有效强化学习的若干潜在方向。

Although DeepSeekMath achieves impressive scores on quantitative reasoning benchmarks, its capability on geometry and theorem-proof are relatively weaker than closed models. For instance, in our dry run, the model cannot handle problems related to triangles and ellipses, which may indicate data selection bias in pre-training and fine-tuning. In addition, restricted by the model scale, DeepSeekMath is worse than GPT-4 on few-shot capability. GPT-4 could improve its performance with few-shot inputs, while DeepSeekMath shows similar performance in zero-shot and few-shot evaluation. In the future, we will further improve our engineered data selection pipeline to construct more high-quality pre-trained corpus. In addition, we will explore the potential directions above for more effective reinforcement learning of LLMs.

尽管 DeepSeekMath 在定量推理基准上取得了令人瞩目的分数，但其几何与定理证明能力仍相对弱于闭源模型。 例如，在我们的试运行中，该模型无法处理与三角形和椭圆有关的问题，这可能表明预训练和微调中存在数据选择偏差。 此外，受模型规模限制，DeepSeekMath 的少样本能力弱于 GPT-4。 GPT-4 可以通过少样本输入提高性能，而 DeepSeekMath 在零样本与少样本评估中表现相近。 未来，我们将进一步改进工程化数据选择流水线，以构建更高质量的预训练语料库。 此外，我们将探索上述潜在方向，以实现更有效的 LLM 强化学习。