---

DeepRAG: Thinking to Retrieve Step by Step for Large Language Models

LLMRAGReasoning中国科学院软件研究所中国科学院大学腾讯微信 AI

Guan X, Zeng J, Meng F, et al. DeepRAG: thinking to retrieve step by step for large language models[PP/OL]. V2. arXiv (2025-06-08)[2026-11-04]. DOI:10.48550/arXiv.2502.01142.

https://arxiv.org/abs/2502.01142v2

https://github.com/gxy-gxy/DeepRAG

---

DeepRAG：面向大语言模型的逐步思考与检索

Abstract

Large Language Models (LLMs) have shown remarkable reasoning capabilities, while their practical applications are limited by severe factual hallucinations due to limitations in the timeliness, accuracy, and comprehensiveness of their parametric knowledge. Meanwhile, enhancing retrieval-augmented generation (RAG) with reasoning remains challenging due to ineffective task decomposition and redundant retrieval, which can introduce noise and degrade response quality. In this paper, we propose DeepRAG, a framework that models retrieval-augmented reasoning as a Markov Decision Process (MDP), enabling reasonable and adaptive retrieval. By iteratively decomposing queries, DeepRAG dynamically determines whether to retrieve external knowledge or rely on parametric reasoning at each step. Experiments show that DeepRAG improves retrieval efficiency and boosts answer accuracy by 26.4%, demonstrating its effectiveness in enhancing retrieval-augmented reasoning.

大语言模型（LLM）展现出突出的推理能力，但由于参数化知识在时效性、准确性和全面性方面存在局限，严重的事实性幻觉限制了它们的实际应用。 与此同时，由于任务分解效果不佳和冗余检索可能引入噪声、降低回答质量，利用推理来增强检索增强生成（RAG）仍然面临挑战。 本文提出 DeepRAG 框架，将检索增强推理建模为马尔可夫决策过程（MDP），从而实现合理且自适应的检索。 通过迭代分解查询，DeepRAG 在每一步动态判断是检索外部知识，还是依赖参数化知识进行推理。 实验表明，DeepRAG 提高了检索效率，并将答案准确率提升了 26.4%，证明了它在增强检索增强推理方面的有效性。

1. Introduction

Large Language Models (LLMs) have shown considerable promise in reasoning. Nevertheless, their limitations in capacity and capabilities result in significant issues with factual hallucinations, stemming from challenges related to the timeliness, accuracy, and comprehensiveness of their parametric knowledge. To mitigate these problems, Retrieval-Augmented Generation (RAG) has been introduced as a promising approach. By incorporating relevant information from knowledge bases or search engines, RAG enhances the factual accuracy of model responses.

大语言模型（LLM）在推理方面展现出可观潜力。 然而，其容量和能力的局限导致了显著的事实性幻觉问题，这些问题源于参数化知识在时效性、准确性和全面性方面面临的挑战。 为缓解这些问题，检索增强生成（RAG）被提出，成为一种有前景的方法。 通过引入来自知识库或搜索引擎的相关信息，RAG 能够提高模型回答的事实准确性。

人类思考过程与 DeepRAG 的对应关系。具体而言，检索叙事 根据先前内容生成寻求补充信息的子查询，以保证流程的结构性；原子决策 则针对每个子查询，动态判断是检索外部知识，还是仅依赖参数化知识。

图1：人类思考过程与 DeepRAG 的对应关系。具体而言，检索叙事 根据先前内容生成寻求补充信息的子查询，以保证流程的结构性；原子决策 则针对每个子查询，动态判断是检索外部知识，还是仅依赖参数化知识。

However, enhancing RAG with reasoning still poses several challenges. One significant issue is that complex queries often necessitate multi-step decomposition to establish a coherent reasoning process. Iterative retrieval has been proposed as a solution to continuously update retrieval results, addressing the dynamic information needs that arise during the generation process. Despite this, LLMs frequently struggle to generate precise and atomic subqueries, which are essential for more effective retrieval and question decomposition.

然而，利用推理来增强 RAG 仍面临若干挑战。 一个重要问题是，复杂查询往往需要经过多步分解，才能建立连贯的推理过程。 迭代检索被提出作为一种解决方案，通过持续更新检索结果，满足生成过程中不断变化的信息需求。 尽管如此，大语言模型仍经常难以生成精确且具有原子性的子查询，而这对于更有效的检索和问题分解至关重要。

From the perspective of RAG, iterative retrieval should ideally generate the next atomic query based on the current question and the available information in an adaptive manner. As illustrated in Figure 1, the process flows logically from one step to the next. Specifically, the goal of finding each movie's runtime in steps 2-4 is derived from step 1's identification of the three titles of the Lord of the Rings series.

从 RAG 的角度看，理想的迭代检索应根据当前问题和已有信息，自适应地生成下一个原子查询。 如图 1所示，这一过程应当逐步形成符合逻辑的衔接。 具体而言，步骤 2–4 中查找每部电影时长的目标，源自 步骤 1 对《指环王》系列三部电影片名的识别。

Additionally, retrieval is not always essential. Some queries depend on external knowledge (steps 2-4), while others can be addressed through the reasoning capabilities of the LLM alone (step 5 requires summarizing previous information). Moreover, LLMs have shown the ability to function as knowledge bases in their own right (such as in step 1, where the three movie titles are widely known). Unnecessary retrieval can be redundant and may introduce noise, and degrade the quality of generated responses.

此外，检索并非总是必要的。 一些查询依赖外部知识（步骤 2–4），而另一些查询仅凭大语言模型的推理能力即可解决（步骤 5 需要汇总之前的信息）。 此外，大语言模型本身也展现出充当知识库的能力，例如 步骤 1 中的三个电影片名就是广为人知的知识。 不必要的检索可能形成冗余、引入噪声，并降低生成回答的质量。

To tackle these issues, we introduce DeepRAG, a new framework inspired by how humans search the Internet based on demand. This framework aims to enhance reasoning capabilities in retrieval-augmented generation by modeling the process as a Markov Decision Process. DeepRAG incorporates two main components: retrieval narrative and atomic decisions, which together create a strategic and adaptive retrieval system. As depicted in Figure 1, the retrieval narrative ensures a structured workflow by generating subqueries that seek additional information based on previous content. For each subquery, atomic decisions dynamically determines whether to retrieve external knowledge or rely solely on the LLM's parametric knowledge.

为应对这些问题，我们提出 DeepRAG，一个受人类按需搜索互联网的方式启发的新框架。 该框架通过将过程建模为马尔可夫决策过程，旨在增强检索增强生成中的推理能力。 DeepRAG 包含两个主要组成部分：检索叙事（retrieval narrative） 和 原子决策（atomic decisions），二者共同构成一个具有策略性和自适应性的检索系统。 如图 1所示，检索叙事 根据先前内容生成寻求补充信息的子查询，从而保证流程的结构性。 对于每个子查询，原子决策 动态判断是检索外部知识，还是仅依赖大语言模型的参数化知识。

As illustrated in Figure 2, our framework consists of three components:

如图 2所示，我们的框架包含三个组成部分：

1. Binary Tree Search, which constructs a binary tree for each subquery related to the given question, exploring paths based on either parametric knowledge or external knowledge.
2. Imitation Learning, which extracts the reasoning process that leads to the correct final answer with minimal retrieval cost based on Binary Tree Search, enabling the model to learn the pattern of “subquery generation – atomic decision – intermediate answer”.
3. Chain of Calibration, which calibrates the LLM's internal knowledge by refining each atomic decision, enabling it to make accurate atomic decisions about the necessity of retrieval.

1. 二叉树搜索：为给定问题相关的每个子查询构建二叉树，探索基于参数化知识或外部知识的路径。
2. 模仿学习：基于二叉树搜索，提取以最小检索成本得到正确最终答案的推理过程，使模型学习“子查询生成—原子决策—中间答案”的模式。
3. 校准链（Chain of Calibration）：通过细化每个原子决策来校准大语言模型的内部知识，使其能够就检索的必要性作出准确的 原子决策。

By explicitly enhancing the LLM's ability to recognize its own knowledge limits, we can train any model in an end-to-end manner, allowing it to dynamically decide when and what to retrieve.

通过显式增强大语言模型识别自身知识局限的能力，我们可以端到端地训练任意模型，使其动态决定何时检索以及检索什么。

We validate the effectiveness of DeepRAG across in-distribution, out-of-distribution, time-sensitive, and heterogeneous knowledge base datasets. Experimental results show that DeepRAG significantly outperforms existing methods, achieving a 21.99% increase in accuracy while also enhancing retrieval efficiency. Further analysis indicates that DeepRAG demonstrates a stronger correlation between its retrieval decisions and parametric knowledge, suggesting more effective calibration of knowledge boundaries.

我们在分布内、分布外、时间敏感以及异构知识库数据集上验证了 DeepRAG 的有效性。 实验结果表明，DeepRAG 显著优于现有方法，准确率提升 21.99%，同时也提高了检索效率。 进一步分析表明，DeepRAG 的检索决策与参数化知识之间具有更强的相关性，说明它能够更有效地校准知识边界。

DeepRAG 概览。框架包含三个步骤：（1）二叉树搜索；（2）模仿学习；（3）校准链。给定一组监督数据集，我们首先通过二叉树搜索合成模仿学习数据，使模型学习有效的检索模式。随后，再通过二叉树搜索生成偏好数据，进一步校准大语言模型对自身知识边界的认识。

图2：DeepRAG 概览。框架包含三个步骤：（1）二叉树搜索；（2）模仿学习；（3）校准链。给定一组监督数据集，我们首先通过二叉树搜索合成模仿学习数据，使模型学习有效的检索模式。随后，再通过二叉树搜索生成偏好数据，进一步校准大语言模型对自身知识边界的认识。

2. Related Work

Adaptive Retrieval-Augmented Generation Existing adaptive RAG approaches can be broadly categorized into three types: classifier-based methods requiring additional linear head training for retrieval decisions, confidence-based methods relying heavily on threshold-dependent uncertainty metrics, and LLM-based methods generating retrieval decisions but often fail to accurately recognize their knowledge boundaries, making it unreliable to delegate retrieval timing decisions to the model. Our method leverages the inherent generative capabilities of LLMs to explore knowledge boundaries in RAG settings. This design maintains the model's native generation abilities while eliminating the need for additional parameters or unreliable uncertainty metrics.

自适应检索增强生成 现有自适应 RAG 方法大致可分为三类：基于分类器的方法，需要额外训练线性头来作出检索决策；基于置信度的方法，高度依赖受阈值影响的不确定性指标；以及基于大语言模型的方法，通过生成来作出检索决策，但往往无法准确识别自身知识边界，因此将检索时机的决策交给模型并不可靠。 我们的方法利用大语言模型自身的生成能力，探索 RAG 场景下的知识边界。 这一设计保留了模型原生的生成能力，同时不需要额外参数或不可靠的不确定性指标。

Reasoning in Retrieval-Augmented Generation Recent advances in RAG have increasingly emphasized the integration of reasoning capabilities. Search-o1 incorporates retrieval into inference to build an agentic system, while its application is limited to reasoning models. Self-RAG and Auto-RAG enhance reasoning through automatic data synthesis within retrieval-augmented frameworks, while AirRAG combines Monte Carlo Tree Search with self-consistency techniques. These methods, however, often depend heavily on extensive retrieval or sampling overhead. More recent developments have explored reinforcement learning to enhance retrieval quality, while these methods generally overlook retrieval efficiency in their reward function. In contrast, DeepRAG offers a flexible, end-to-end solution that enables arbitrary models to retrieve information step by step as needed, based on their evolving reasoning process.

检索增强生成中的推理 RAG 的近期进展日益重视推理能力的融入。 Search-o1 将检索引入推理过程来构建智能体系统，但其应用仅限于推理模型。 Self-RAG 和 Auto-RAG 通过在检索增强框架中自动合成数据来增强推理，而 AirRAG 则将蒙特卡洛树搜索与自一致性技术相结合。 然而，这些方法往往高度依赖大量检索或采样开销。 更新近的研究探索了利用强化学习提升检索质量，但这些方法通常在奖励函数中忽略了检索效率。 相比之下，DeepRAG 提供了一种灵活的端到端方案，使任意模型都能够根据不断推进的推理过程，按需逐步检索信息。

Knowledge Boundary LLMs struggle to accurately distinguish between what they know and what they don't know. Additional fine-tuning or precise probing is typically required to calibrate the model's cognition. Our approach explores knowledge boundaries in RAG settings.

知识边界 大语言模型难以准确区分自己知道什么、不知道什么。 通常需要额外微调或精确探测，才能校准模型的认知。 我们的方法探索 RAG 场景下的知识边界。

3. Thinking to Retrieve Step by Step

In this section, we present our proposed method, DeepRAG. At its core, DeepRAG models the process of question decomposition, atomic decisions, and final answer generation as a Markov Decision Process. Given a set of supervised datasets, we first use binary tree search to synthesize data for imitation learning, allowing the model to learn effective retrieval patterns. Next, we employ binary tree search to generate preference data, further calibrating the LLM's awareness of its knowledge boundaries. In the following subsections, we provide a detailed description of each component of DeepRAG.

本节介绍我们提出的方法 DeepRAG。 DeepRAG 的核心是将问题分解、原子决策和最终答案生成的过程建模为马尔可夫决策过程。 给定一组监督数据集，我们首先使用二叉树搜索合成模仿学习数据，让模型学习有效的检索模式。 随后，我们使用二叉树搜索生成偏好数据，进一步校准大语言模型对自身知识边界的认识。 下面各小节将详细介绍 DeepRAG 的各个组成部分。

3.1. Overview of the MDP Modeling

We formalize the step-by-step reasoning process of retrieval-augmented generation as a Markov Decision Process, represented by the tuple $(\mathcal{S},\mathcal{A},P,R)$, where $\mathcal{S}$ denotes the set of states, $\mathcal{A}$ represents the set of actions, $P$ defines the transition dynamics, and $R$ specifies the reward function.

我们将检索增强生成的逐步推理过程形式化为马尔可夫决策过程，用元组 $(\mathcal{S},\mathcal{A},P,R)$ 表示，其中 $\mathcal{S}$ 表示状态集合，$\mathcal{A}$ 表示动作集合，$P$ 定义转移动态，$R$ 指定奖励函数。

States. At each step $t$, the state $s_t\in \mathcal{S}$ represents the partial solution to the original question. We denote

状态。 在每一步 $t$，状态 $s_t\in \mathcal{S}$ 表示原始问题的部分解。 我们将其记为

$$s_t=[x,(q_1,r_1),\dots ,(q_t,r_t)],$$

where $x$ is the input question, $q_i$ refers to the $i$-th subquery, and $r_i$ refers to the $i$-th intermediate answer (and any retrieved documents based on $q_i$).

其中，$x$ 是输入问题，$q_i$ 表示第 $i$ 个子查询，$r_i$ 表示第 $i$ 个中间答案，以及基于 $q_i$ 检索到的任何文档。

Actions. At state $s_t$, the model selects an action $a_{t+1}=({\sigma }_{t+1},{\delta }_{t+1})\in \mathcal{A}$, which consists of two sub-decisions:

动作。 在状态 $s_t$ 下，模型选择动作 $a_{t+1}=({\sigma }_{t+1},{\delta }_{t+1})\in \mathcal{A}$，其中包含两个子决策：

1. Termination decision: Given the partial solution $s_t$, the model makes a binary decision ${\sigma }_{t+1}\in \{\mathtt{\text{continue}},\mathtt{\text{terminate}}\}$ to determine whether to proceed with generating the next subquery $q_{t+1}$ or finalize the answer $o$.
2. Atomic decision: For each subquery $q_{t+1}$, the model decides whether to retrieve external knowledge or rely solely on its parametric knowledge. Formally, this decision is represented as ${\delta }_{t+1}\in \{\mathtt{\text{retrieve}},\mathtt{\text{parametric}}\}$.

1. 终止决策：给定部分解 $s_t$，模型作出二元决策 ${\sigma }_{t+1}\in \{\mathtt{\text{continue}},\mathtt{\text{terminate}}\}$，以判断是继续生成下一个子查询 $q_{t+1}$，还是确定最终答案 $o$。
2. 原子决策：对于每个子查询 $q_{t+1}$，模型决定是检索外部知识，还是仅依赖其参数化知识。形式化地，这一决策表示为 ${\delta }_{t+1}\in \{\mathtt{\text{retrieve}},\mathtt{\text{parametric}}\}$。

Transitions. After executing the action $a_{t+1}=({\sigma }_{t+1},{\delta }_{t+1})$ in state $s_t$, the environment updates the state to $s_{t+1}$ based on transition dynamics $P$.

转移。 在状态 $s_t$ 下执行动作 $a_{t+1}=({\sigma }_{t+1},{\delta }_{t+1})$ 后，环境依据转移动态 $P$，将状态更新为 $s_{t+1}$。

Specifically, if ${\sigma }_{t+1}=\mathtt{\text{terminate}}$, the process concludes by generating the final answer $o$, resulting in the terminal state

具体而言，如果 ${\sigma }_{t+1}=\mathtt{\text{terminate}}$，则生成最终答案 $o$ 并结束这一过程，得到终止状态

$$\begin{array}{rl}{s}_{t+1}& =[x,(q_1,r_1),\\ & \dots ,(q_t,r_t),o].\end{array}$$

Otherwise, it generates the next subquery $q_{t+1}$.

否则，生成下一个子查询 $q_{t+1}$。

If ${\delta }_{t+1}=\mathtt{\text{retrieve}}$, the model retrieves documents $d_{t+1}$ and generates an intermediate answer $i{a}_{t+1}$ for subquery $q_{t+1}$. Otherwise, it relies on parametric knowledge to generate the intermediate answer. The response $r_{t+1}$ is set as $[d_{t+1},i{a}_{t+1}]$ (if retrieved) or $i{a}_{t+1}$ (if not). The updated state is

如果 ${\delta }_{t+1}=\mathtt{\text{retrieve}}$，模型会检索文档 $d_{t+1}$，并为子查询 $q_{t+1}$ 生成中间答案 $i{a}_{t+1}$。 否则，模型依赖参数化知识生成中间答案。 回答 $r_{t+1}$ 在进行了检索时设为 $[d_{t+1},i{a}_{t+1}]$，未检索时则设为 $i{a}_{t+1}$。 更新后的状态为

$$\begin{array}{rl}{s}_{t+1}& =[x,(q_1,r_1),\\ & \dots ,(q_{t+1},r_{t+1})].\end{array}$$

Rewards. The reward function evaluates the state based on answer correctness and retrieval cost, applied only after generating the final answer $o$. Formally,

奖励。 奖励函数根据答案正确性和检索成本评估状态，且仅在生成最终答案 $o$ 之后应用。 形式化地，

$$R(s_{t+1}=s_t+[o])=-C(o)\times T(s_t),$$

where $C(o)$ indicates correctness ($1$ if correct, $\mathrm{\infty }$ otherwise), and $T(s_t)$ represents the total retrieval cost in state $s_t$. Therefore, this reward prioritizes answer correctness while encouraging the model to reduce retrieval cost as much as possible.

其中，$C(o)$ 表示正确性，正确时为 $1$，否则为 $\mathrm{\infty }$；$T(s_t)$ 表示状态 $s_t$ 下的总检索成本。 因此，这一奖励优先保证答案正确性，同时鼓励模型尽可能降低检索成本。

3.2. Binary Tree Search

Building on this formulation, LLM iteratively decomposes a given question into subqueries, each derived from previously acquired information. The detailed generation instruction is outlined in Appendix A.2, with the answer format below.

基于上述形式化描述，大语言模型将给定问题迭代分解为子查询，每个子查询都由先前获取的信息推导而来。 详细的生成指令见附录 A.2，答案格式如下。

Answer format

Question: <Question>
Follow up: <Subquery1>
Let's search the question in Wikipedia.
Context: <Paragraph Text>
Intermediate answer: <Intremediate Answer1>
Follow up: <Subquery2>
Intermediate answer: <Intermediate Answer2>
......
So the final answer is: <Answer>

Then, we implement a binary tree search to construct reasoning paths that integrate different retrieval strategies for each subquery. As illustrated in Figure 2, given a question, the model generates the $i$-th subquery and explores two answering strategies: directly leveraging parametric knowledge (blue node) or retrieving external documents (green node). Therefore, we can construct a binary tree for each subquery related to the given question, exploring paths based on either parametric knowledge or external knowledge.

随后，我们通过二叉树搜索构建推理路径，为每个子查询整合不同的检索策略。 如图 2所示，给定一个问题，模型生成第 $i$ 个子查询，并探索两种回答策略：直接利用参数化知识（蓝色节点），或者检索外部文档（绿色节点）。 因此，我们可以为给定问题相关的每个子查询构建二叉树，探索基于参数化知识或外部知识的路径。

3.3. Imitation Learning

We present an algorithm that leverages binary trees to identify the optimal reasoning process that leads to the correct final answer while minimizing retrieval costs, corresponding to the highest reward as defined in Section 3.1. Based on the synthesized optimal reasoning data, we fine-tune the model to improve its termination and atomic decisions while enhancing its query decomposition capabilities and generating faithful intermediate answers.

我们提出一种利用二叉树的算法，寻找能够得到正确最终答案、同时使检索成本最小的最优推理过程，这对应于第 3.1 节定义的最高奖励。 基于合成的最优推理数据，我们微调模型，改善其终止决策和原子决策，同时增强查询分解能力，并使其生成忠实的中间答案。

Synthesizing Data As shown in Alg. 1, we employ a priority queue to maintain reasoning trajectories based on their retrieval costs. This allows us to efficiently explore potential reasoning paths by iteratively constructing and evaluating them until either finding a correct answer or exhausting all viable options within specified constraints. For instances where no correct answer can be obtained after exhausting all options, we discard them.

数据合成 如算法 1所示，我们使用优先队列，根据检索成本维护推理轨迹。 这样便能通过迭代构建和评估推理路径，高效探索潜在路径，直至找到正确答案，或在指定约束下穷尽所有可行选项。 对于穷尽所有选项后仍无法得到正确答案的样本，我们将其丢弃。

Through the synthesis process above, the training dataset obtained contains an adaptive reasoning process, which can be used to facilitate arbitrary LLMs in enhancing the RAG capabilities.

通过上述合成过程，得到的训练数据集包含自适应推理过程，可用于帮助任意大语言模型增强 RAG 能力。

Algorithm 1: Data Construction for Stage I

Require: Question x, answer y, language model ℳ, Retriever ℛ, max history length T

Ensure: Optimal reasoning process s* or null

1. Initialize priority queue 𝒫𝒬 ← {([x], 0)} ▷ (trajectory, retrieval count)
2. while 𝒫𝒬 is not empty do
3. (h, r) ← 𝒫𝒬.dequeue() ▷ Get trajectory with lowest retrieval count
4. q ← ℳ(h) ▷ Subquery Generation
5. if ShouldAnswer(q) or length(h) > T then
6. o ← ℳ(h, q) ▷ Final answer
7. if IsEqual(o, y) then return h
8. else
9. a ← ℳ(h, q) ▷ Direct answer
10. 𝒫𝒬.enqueue(([h, (q, a)], r))
11. d ← ℛ(q) ▷ Retrieve document
12. a ← ℳ(h, q, d) ▷ Retrieved answer
13. 𝒫𝒬.enqueue(([h, (q, (d, a))], r+1))
14. return null

Training Objective We implement a masked loss function for the retrieved documents to prevent the model from learning irrelevant or noisy text that could negatively impact its performance. The detailed objective is shown in Appendix B.1.

训练目标 我们采用对检索文档进行掩码处理的损失函数，防止模型学习可能对性能产生负面影响的无关或带噪文本。 详细目标见附录 B.1。

3.4. Chain of Calibration

Building on the markov process in Section 3.1, we identify four key optimization aspects for DeepRAG: termination and atomic decisions, query decomposition, and intermediate answer generation. Unlike the others, atomic decisions require the model to recognize its own knowledge boundaries to make precise judgments.

基于第 3.1 节中的马尔可夫过程，我们确定了 DeepRAG 的四个关键优化方面：终止决策、原子决策、查询分解和中间答案生成。 与其他方面不同，原子决策 要求模型识别自身的知识边界，才能作出准确判断。

We propose a method that dynamically optimizes atomic decisions for each subquery, rather than training LLMs on complete reasoning paths. Our approach consists of two key components:

我们提出一种针对每个子查询动态优化原子决策的方法，而不是在完整推理路径上训练大语言模型。 这一方法包含两个关键组成部分：

1. synthesizing preference data to determine when retrieval is necessary, and
2. fine-tuning the LLM with this data using Chain of Calibration training to enhance its ability to make informed atomic decisions based on its internal knowledge boundaries.

1. 合成偏好数据，以确定何时需要检索；
2. 使用这些数据，通过校准链训练来微调大语言模型，增强其根据自身知识边界作出有依据的原子决策的能力。

Synthesizing Preference Data First, we identify an optimal path with minimal retrieval based on Alg. 1 using the model trained in Stage I. This provides the optimal atomic decision for each subquery, determining whether retrieval is necessary. From this path, we construct preference pairs for each subquery to indicate the preferred retrieval choice. For example, in Figure 2, the optimal path may suggest answering the first subquery using parametric knowledge while requiring document retrieval for the second. Accordingly, we generate preference pairs favoring parametric knowledge for the first subquery and retrieval for the second. This process enables LLMs to learn when to retrieve external information, thereby improving its ability to maximize the use of parametric knowledge and reducing unnecessary retrievals.

偏好数据合成 首先，使用阶段 I 训练得到的模型，基于算法 1，找出检索次数最少的最优路径。 这为每个子查询提供最优原子决策，确定是否需要检索。 根据这条路径，我们为每个子查询构建偏好对，以指明更优的检索选择。 例如，在图 2中，最优路径可能建议使用参数化知识回答第一个子查询，而第二个子查询则需要检索文档。 相应地，我们生成的偏好对，会对第一个子查询偏好参数化知识，对第二个子查询偏好检索。 这一过程使大语言模型学会何时检索外部信息，从而增强其充分利用参数化知识的能力，并减少不必要的检索。

Chain of Calibration Objective We fine-tune the LLM using a Chain of Calibration objective on our synthesized preference data. Given the $i$-th subquery and a state

校准链目标 我们在合成的偏好数据上，使用校准链目标微调大语言模型。 给定第 $i$ 个子查询以及状态

$$s_i=[x,(q_1,r_1),\cdots ,(q_{i-1},r_{i-1})],$$

we have two distince intermediate answer $r_i^1=a_i^1$ and $r_i^2=(d_i,a_i^2)$. Based on the synthesis process above, we can tag which $r_i$ is preferred and optimize it. The detailed equation is shown in Appendix B.2.

我们有两个不同的中间答案 $r_i^1=a_i^1$ 和 $r_i^2=(d_i,a_i^2)$。 基于上述合成过程，可以标注哪一个 $r_i$ 更受偏好，并对其进行优化。 详细公式见附录 B.2。

4. Experiment

4.1. Datasets

We use six open-domain QA datasets for our experiments. We treat training datasets as in-distribution, and unseen ones as out-of-distribution. The in-distribution datasets include HotpotQA, and 2WikMultihopQA, and the out-of-distribution datasets consist of CAG, PopQA, WebQuestions, and MuSiQue. Specifically, we employ the time-sensitive subset of CAG to evaluate temporal reasoning capabilities. Furthermore, WebQuestions is built upon Freebase to assess model robustness when information may be absent from the knowledge base.

我们的实验使用六个开放域问答数据集。 将训练中使用的数据集视为 分布内 数据集，将未见过的数据集视为 分布外 数据集。 分布内数据集包括 HotpotQA 和 2WikMultihopQA，分布外数据集包括 CAG、PopQA、WebQuestions 和 MuSiQue。 具体而言，我们使用 CAG 的时间敏感子集来评估时间推理能力。 此外，WebQuestions 基于 Freebase 构建，用于评估当知识库可能缺少相关信息时模型的鲁棒性。

4.2. Baselines

We use the following baselines to evaluate the performance: CoT and CoT*, which employ 8-shot examples extracted from the training dataset. The asterisk (*) indicates the model output was trained using the same data employed for training the DeepRAG. CoT-Retrieve and CoT-Retrieve* augment the eight examples in the context with retrieved relevant documents based on the query. IterDRAG refers to decomposing the question and answer step by step based on in-context learning. AutoRAG uses trained models to iteratively decompose questions and retrieve relevant documents for answering.

我们使用以下基线来评估性能：CoT 和 CoT*，二者采用从训练数据集中提取的 8 个少样本示例。 星号（*）表示使用与训练 DeepRAG 相同的数据，对模型输出进行了训练。 CoT-Retrieve 和 CoT-Retrieve* 根据查询检索相关文档，用这些文档增强上下文中的八个示例。 IterDRAG 指基于上下文学习，逐步分解问题并作答。 AutoRAG 使用经过训练的模型，迭代分解问题并检索相关文档来回答。

Search-o1 leverages special tokens to prompt reasoning models to autonomously invoke retrieval as needed. UAR employs a trained classifier to determine when to retrieve. FLARE and DRAGIN are confidence-based method that decide the timing of retrieval based on token importance and uncertainty. TAARE allows the LLM itself to determine when retrieval is needed.

Search-o1 利用特殊 token，提示推理模型根据需要自主调用检索。 UAR 使用训练过的分类器来判断何时检索。 FLARE 和 DRAGIN 是基于置信度的方法，根据 token 重要性和不确定性决定检索时机。 TAARE 让大语言模型自行判断何时需要检索。

表1：DeepRAG 与其他基线在五个基准上的总体实验结果。各数据集的最佳与次佳得分分别以 粗体 和下划线标出。DeepRAG-Imi（阶段 I）和 DeepRAG（阶段 II）在所有测试场景中均表现出优于现有方法的性能。

Types	Methods	in-distribution				out-of-distribution								Avg
		Hotpot QA		2Wiki MultihopQA		CAG		PopQA		Web Question		MuSiQue
		EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
Llama-3-8B
Reasoning	CoT	27.20	37.75	28.20	34.85	7.17	10.41	21.20	25.33	25.20	40.56	13.70	22.97	24.54
	CoT-Retrieve	34.90	46.85	35.80	43.41	55.45	64.08	32.80	45.87	22.90	39.22	19.10	28.18	39.05
	CoT*	21.80	31.69	25.60	30.89	5.30	7.58	23.10	25.31	26.80	40.20	4.80	13.85	21.41
	CoT-Retrieve*	22.50	32.15	23.70	29.21	44.86	55.69	38.70	45.64	17.60	29.20	5.70	11.60	29.71
	IterDRAG	23.20	30.95	19.60	24.80	38.32	46.18	22.70	34.53	15.90	26.79	12.40	17.75	26.09
	Auto-RAG	25.80	36.09	23.00	30.09	49.22	59.61	27.80	42.02	17.40	32.94	19.10	28.33	32.62
	Search-o1	14.80	24.08	22.20	27.10	3.43	6.61	10.30	13.54	15.30	29.60	5.40	11.98	15.36
Adaptive	FLARE	23.80	32.88	30.30	37.45	34.89	43.45	28.80	40.61	28.80	40.61	14.50	23.57	31.64
	DRAGIN	27.60	38.05	29.10	35.68	4.05	7.18	22.60	28.53	21.20	38.72	11.80	19.97	23.71
	UAR	29.70	40.66	34.80	42.40	52.96	61.53	33.00	45.95	22.70	39.10	19.10	28.38	37.52
	TAARE	30.60	41.43	35.20	42.85	52.96	61.59	33.20	46.01	23.40	39.56	18.60	27.55	37.75
Ours	DeepRAG-Imi	35.10	46.59	47.20	52.33	50.47	59.55	43.60	48.50	30.00	41.76	22.30	30.46	42.32
	DeepRAG	40.70	51.54	48.10	53.25	52.96	61.92	42.50	47.80	32.70	45.24	22.50	30.40	44.13
Qwen-2.5-7B
Resaoning	CoT	18.90	27.81	23.40	28.97	3.12	5.71	15.20	19.20	18.30	34.86	5.60	13.12	17.85
	CoT-Retreive	24.90	34.78	18.60	23.44	41.43	51.47	27.30	41.20	15.10	29.84	6.70	15.10	27.49
	CoT*	17.60	26.15	25.10	29.62	3.12	5.62	7.90	11.06	15.60	32.45	4.70	13.40	16.03
	CoT-Retrieve*	23.40	32.29	22.40	27.51	43.30	54.51	26.60	35.46	13.80	25.60	6.20	12.85	26.99
	IterDRAG	13.70	26.84	9.30	20.47	21.81	39.59	18.00	31.44	12.50	26.95	9.20	17.25	20.59
	Search-o1	11.60	16.95	22.00	25.02	3.43	4.78	4.40	7.61	7.70	19.97	2.10	7.48	11.09
Adaptive	FLARE	23.40	32.06	21.80	26.51	34.89	42.62	19.00	28.24	16.10	31.89	8.40	15.15	25.00
	DRAGIN	16.70	24.60	12.40	16.76	3.43	5.45	12.00	15.80	17.40	32.43	4.20	7.98	14.10
	UAR	24.50	34.22	23.90	28.20	34.89	43.92	27.00	40.47	16.60	32.28	7.10	15.62	27.39
	TAARE	25.30	35.03	21.30	25.67	40.81	50.78	27.00	40.92	18.20	33.14	6.90	15.46	28.38
Ours	DeepRAG-Imi	30.40	39.44	32.00	38.32	47.98	56.99	37.50	40.72	23.90	38.62	16.50	24.67	35.59
	DeepRAG	32.10	41.14	40.40	44.87	51.09	59.76	40.60	43.19	24.20	38.83	19.50	32.35	39.00
Qwen-2.5-32B
Reasoning	CoT	20.6	30.62	24.4	30.94	3.12	5.42	10.9	14.45	9.7	26	9.5	18.26	16.99
	CoT-Retrieve	28.6	39.43	27.9	36.73	39.56	49.97	33.8	45.91	17.2	34.15	12.9	21.98	32.34
	Iter-DRAG	22.9	38.26	19.6	35.70	33.02	45.61	20.3	33.2	13.3	27.57	17.6	27.8	27.91
	Search-o1	34	45.64	29.1	35.12	19	24.35	23.1	30.69	17.9	35.11	16.4	25.6	28.00
Ours	DeepRAG	36.2	46.90	46.3	50.50	52.02	61.42	46.3	49.2	28.1	43.27	19.60	27.47	42.27

DeepRAG 与其他基线在五个基准上的总体实验结果。各数据集的最佳与次佳得分分别以 粗体 和下划线标出。DeepRAG-Imi（阶段 I）和 DeepRAG（阶段 II）在所有测试场景中均表现出优于现有方法的性能。

4.3. Implementation Details

We train our target model on two QA datasets: HotpotQA and 2WikiMultihopQA. For imitation learning, we randomly sample 4,000 examples from each dataset. To enhance the model's question decomposition and context-based generation capabilities, we employ Qwen-2.5-72B to generate the gray (query decomposition) and green nodes (retrieved answers) in Figure 2, and use the target model to generate the blue nodes (parametric answers) for data synthesis. For chain of calibration, we sample an additional 1,000 examples from each dataset. The performance is evaluated using Exact Match (EM) and F1 score.

我们在 HotpotQA 和 2WikiMultihopQA 这两个问答数据集上训练目标模型。 对于模仿学习，从每个数据集中随机采样 4,000 个样本。 为增强模型的问题分解和基于上下文生成的能力，我们使用 Qwen-2.5-72B 生成图 2中的灰色节点（查询分解）和绿色节点（检索答案），并使用目标模型生成蓝色节点（参数化答案），用于数据合成。 对于校准链，从每个数据集中额外采样 1,000 个样本。 使用精确匹配（EM）和 F1 分数评估性能。

Following Su et al., we adopt BM25 for retrieval and Wikipedia as knowledge base. For time-sensitive questions in CAG, we utilize the dataset-provided up-to-date passages as knowledge base. We selected Llama-3-8B-Instruct, Qwen-2.5-7B and Qwen-2.5-32B as our target model. To implement Search-o1, we employ the distillation series of DeepSeek-R1, as the method depends on reasoning models.

遵循 Su 等人的方法，我们采用 BM25 进行检索，并使用维基百科作为知识库。 对于 CAG 中的时间敏感问题，使用该数据集提供的最新段落作为知识库。 我们选择 Llama-3-8B-Instruct、Qwen-2.5-7B 和 Qwen-2.5-32B 作为目标模型。 由于 Search-o1 依赖推理模型，因此实现该方法时采用 DeepSeek-R1 蒸馏系列模型。

4.4. Overall Results

The results in Table 1 demonstrate DeepRAG's superior performance and robustness across different scenarios.

表 1中的结果表明，DeepRAG 在不同场景中具有优越的性能和鲁棒性。

DeepRAG demonstrates superior performance across most datasets via thinking to retrieve step by step. Our method consistently outperforms existing approaches across various backbones and model sizes. Compared to reasoning-based and adaptive RAG baselines, DeepRAG outperforms across all datasets, demonstrating the effectiveness of the structured retrieval narrative and on-demand atomic decisions. Specifically, the limited performance of IterDRAG highlights the necessity of learning both query decomposition and faithful answering. Confidence-based methods like FLARE struggle to determine the optimal retrieval timing due to their reliance on unstable, predefined metrics. Moreover, we observe that confidence-based methods suffer from instability, as their performance is highly sensitive to threshold selection. Meanwhile, iterative retrieval methods like Auto-RAG often fall into continuous retrieval loops when no highly relevant information is found.

通过逐步思考来检索，DeepRAG 在大多数数据集上展现出优越性能。 我们的方法在不同骨干模型和模型规模上均持续优于现有方法。 与基于推理和自适应 RAG 的基线相比，DeepRAG 在所有数据集上都表现更优，证明了结构化 检索叙事 和按需 原子决策 的有效性。 具体而言，IterDRAG 的有限表现凸显了学习查询分解与忠实回答这两种能力的必要性。 FLARE 等基于置信度的方法，依赖不稳定的预定义指标，因此难以确定最优检索时机。 此外，我们观察到，基于置信度的方法存在不稳定性，其性能对阈值选择高度敏感。 与此同时，Auto-RAG 等迭代检索方法在找不到高度相关的信息时，往往会陷入持续检索循环。

DeepRAG exhibits remarkable generalization capabilities and robustness in time-sensitive and out-of-distribution settings. In the time-sensitive dataset CAG, DeepRAG performs well compared to other adaptive and reasoning retrieval methods. It is worth noting that CoT-Retrieve outperforms it on CAG. We attribute this to the core challenge of the time-sensitive setting is to trigger retrieval most of the time. Furthermore, DeepRAG achieves substantial F1 score improvements of 2.63 and 4.57 on PopQA and WebQuestions respectively, even in scenarios where relevant information may be sparse or missing from the knowledge base.

DeepRAG 在时间敏感和分布外设置中展现出突出的泛化能力与鲁棒性。 在时间敏感数据集 CAG 上，与其他自适应和推理检索方法相比，DeepRAG 表现良好。 需要指出的是，CoT-Retrieve 在 CAG 上的表现优于 DeepRAG。 我们认为，这是因为时间敏感设置的核心挑战在于，大多数时候都应触发检索。 此外，即使在知识库中相关信息可能稀缺或缺失的场景下，DeepRAG 在 PopQA 和 WebQuestions 上的 F1 分数仍分别取得了 2.63 和 4.57 的显著提升。

By learning from self-synthesized data, DeepRAG effectively explores knowledge boundaries while minimizing hallucination risks. TAARE often underperforms direct retrieval methods, highlighting the mismatch between its internal knowledge and verbose. Moreover, aggressive fine-tuning approaches like CoT* and CoT-Retrieve* degrade model performance by forcing the model to learn knowledge beyond its knowledge boundaries. DeepRAG carefully preserves model capabilities during fine-tuning by leveraging self-synthesized data, effectively preventing additional hallucination while maintaining performance.

通过学习自行合成的数据，DeepRAG 能够有效探索知识边界，同时尽量降低幻觉风险。 TAARE 的表现往往不及直接检索方法，凸显了其内部知识与言语表达之间的不匹配。 此外，CoT* 和 CoT-Retrieve* 等激进的微调方法，迫使模型学习超出自身知识边界的知识，从而降低了模型性能。 DeepRAG 在微调期间谨慎地保留模型能力，利用自行合成的数据，在保持性能的同时有效防止额外幻觉。

5. Analysis

5.1. Retrieval Efficiency

To evaluate the efficiency of our method, we compare the average number of retrievals on the WebQuestions dataset and report the average computation time per query. The computation time is measured on an H20*8 machine. As shown in Table 2, We have the following observations:

为评估我们的方法的效率，我们比较了 WebQuestions 数据集上的平均检索次数，并报告每个查询的平均计算时间。 计算时间在一台配备 8 张 H20 的机器上测得。 如表 2所示，我们有以下观察结果：

1. DeepRAG can achieve higher accuracy with relatively lower retrieval costs, attributed to its dynamic usage of internal knowledge.
2. Confidence-based approaches demonstrate limited robustness across datasets. For instance, neither FLARE nor DRAGIN triggers retrieval under the default confidence threshold in the WebQuestions dataset.
3. Iterative retrieval-based methods typically require numerous retrieval operations.

1. DeepRAG 能够以相对较低的检索成本取得更高准确率，这归功于它对内部知识的动态利用。
2. 基于置信度的方法在不同数据集上的鲁棒性有限。例如，在 WebQuestions 数据集上，FLARE 和 DRAGIN 在默认置信度阈值下都不会触发检索。
3. 基于迭代检索的方法通常需要执行大量检索操作。

Therefore, efficient adaptive retrieval methods like DeepRAG become crucial for optimizing resource utilization while maintaining performance.

因此，DeepRAG 这类高效的自适应检索方法，对于在保持性能的同时优化资源利用至关重要。

表2：不同方法在 WebQuestions 上的检索频次分析。“Correct”表示模型回答正确的样本的平均检索次数，“Incorrect”表示回答错误的样本的平均检索次数。Time 为每个样本的平均耗时，单位为秒。

Method	EM	Avg. Retrievals			Time
		All	Correct	Incorrect
FLARE	28.80	0.00	0.00	0.00	2.58
DRAGIN	21.20	0.00	0.00	0.00	1.36
UAR	22.70	0.96	0.95	0.97	0.43
TAARE	23.40	0.66	0.65	0.66	0.11
IterDRAG	15.90	2.25	2.16	2.27	1.09
Auto-RAG	17.40	4.52	3.03	2.35	0.71
DeepRAG-Imi	30.00	0.43	0.13	0.56	0.67
DeepRAG	32.70	0.28	0.12	0.36	0.50

不同方法在 WebQuestions 上的检索频次分析。“Correct”表示模型回答正确的样本的平均检索次数，“Incorrect”表示回答错误的样本的平均检索次数。Time 为每个样本的平均耗时，单位为秒。

5.2. Relevance to Parametric Knowledge

In this section, we investigate the relationship between retrieval needs and internal knowledge to demonstrate how effectively atomic decisions explores the knowledge boundary.

本节研究检索需求与内部知识之间的关系，以说明 原子决策 能够多么有效地探索知识边界。

Ideally, models should initiate retrieval for queries beyond their parametric knowledge while utilizing their existing knowledge for familiar queries. We use CoT results as an indicator of whether the model can answer questions using its parametric knowledge. Then, we analyze whether other adaptive retrieval methods align with this pattern of parametric knowledge utilization.

理想情况下，模型应当对超出其参数化知识的查询发起检索，而对熟悉的查询则利用已有知识。 我们以 CoT 的结果作为模型能否使用参数化知识回答问题的指标。 随后，分析其他自适应检索方法是否符合这种参数化知识利用模式。

We report four metrics. F1 score and Accuracy serve as basic performance measures, while balanced accuracy and Matthews Correlation Coefficient(MCC) are employed to account for the class imbalance between retrieval-required and retrieval-not-required cases.

我们报告四项指标。 F1 分数和准确率作为基本性能指标，而平衡准确率与马修斯相关系数（MCC）用于考虑需要检索与不需要检索两类情况之间的类别不平衡。

As shown in Table 3, we find that:

如表 3所示，我们发现：

1. DeepRAG demonstrates superior relevance performance across F1, balanced accuracy, and MCC metrics. This suggests that DeepRAG successfully identifies retrieval necessity by exploring knowledge boundary;
2. While FLARE, DRAGIN, and TAARE exhibit high accuracy scores, their relatively low balanced accuracy and MCC scores suggest they mainly succeed in retrieval-required cases but struggle to properly avoid unnecessary retrievals.

1. DeepRAG 在 F1、平衡准确率和 MCC 指标上均展现出更好的相关性表现。这说明，DeepRAG 通过探索知识边界，成功识别了检索的必要性；
2. 尽管 FLARE、DRAGIN 和 TAARE 具有较高准确率，但它们相对较低的平衡准确率与 MCC 表明，这些方法主要在需要检索的情况下取得成功，却难以妥善避免不必要的检索。

表3：不同自适应检索方法在 2WikiMultihopQA 上的内部知识利用情况分析。

Method	F1	Acc	Balanced Acc	MCC
FLARE	0.000	0.718	0.500	0.000
DRAGIN	0.007	0.709	0.495	-0.045
UAR	0.481	0.756	0.648	0.341
TAARE	0.127	0.712	0.518	0.078
Iter-DRAG	0.000	0.718	0.500	0.000
Auto-RAG	0.000	0.718	0.500	0.000
DeepRAG-Imi	0.580	0.732	0.709	0.393
DeepRAG	0.621	0.749	0.743	0.451

不同自适应检索方法在 2WikiMultihopQA 上的内部知识利用情况分析。

强化学习设置下，在 2WikiMultihopQA 上的实验结果与检索效率。

图3：强化学习设置下，在 2WikiMultihopQA 上的实验结果与检索效率。

不同检索策略的比较分析：仅使用参数化知识（parametric only）或仅使用检索（retrieve only）。

图4：不同检索策略的比较分析：仅使用参数化知识（parametric only）或仅使用检索（retrieve only）。

模仿学习消融实验中的平均得分与检索次数。

图5：模仿学习消融实验中的平均得分与检索次数。

校准链消融实验中的平均得分与检索次数。

图6：校准链消融实验中的平均得分与检索次数。

（a）子查询数量统计。（b）检索次数统计。

图7：（a）子查询数量统计。（b）检索次数统计。

5.3. Effectiveness under RL Setting

Recently, reinforcement learning (RL) has demonstrated remarkable success in enhancing model capabilities across various domains. Building upon this, we further explore the potential of DeepRAG by incorporating reinforcement learning. Specifically, we initialize from DeepRAG-Imi and optimize Stage II using the GRPO objective. The detailed implementation can be found in Appendix C.5, and our code is released in GitHub repository.

近期，强化学习（RL）在提升不同领域的模型能力方面取得了显著成功。 在此基础上，我们进一步引入强化学习，探索 DeepRAG 的潜力。 具体而言，以 DeepRAG-Imi 进行初始化，并使用 GRPO 目标优化阶段 II。 详细实现见附录 C.5，代码已在 GitHub 仓库中发布。

Figure 3 presents the training dynamics of our RL-enhanced model. The results reveal an encouraging trend: as training progresses, the F1 score on 2WikiMultihopQA gradually improves while the average number of retrievals decreases. This demonstrates that our reward design effectively guides the model to achieve better performance with more efficient retrieval behavior.

图 3展示了引入强化学习后的模型训练动态。 结果显示出令人鼓舞的趋势：随着训练推进，2WikiMultihopQA 上的 F1 分数逐步提高，而平均检索次数下降。 这表明，我们的奖励设计有效引导模型以更高效的检索行为取得更好的性能。

5.4. Question Decomposition Effectiveness

We systematically analyze the effectiveness of question decomposition in retrieval narrative. As shown in Figure 7, we present the distribution of subquery counts and retrieval attempts for different questions. Most questions require 3-5 decomposition steps, while retrieval attempts are primarily concentrated within 0-2 rounds. This demonstrates that DeepRAG effectively decomposes questions while minimizing redundant retrieval.

我们系统分析了 检索叙事 中问题分解的有效性。 如图 7所示，我们给出了不同问题的子查询数量和检索尝试次数分布。 大多数问题需要 3–5 步分解，而检索尝试主要集中在 0–2 轮。 这表明，DeepRAG 能够有效分解问题，同时尽量减少冗余检索。

Moreover, we analyze the average counts of WH-words, nouns, verbs, and conjunctions in subqueries, as shown in Figure 8. DeepRAG decomposes atomic queries with fewer pronouns and conjunctions, indicating its concise and effective query decomposition strategy.

此外，我们分析了子查询中 WH 词、名词、动词和连词的平均数量，如图 8所示。 DeepRAG 分解出的原子查询包含更少的代词和连词，说明其查询分解策略简洁而有效。

5.5. Different Inference Strategy

To gain a deep insight into the effectiveness of atomic decision, we evaluate DeepRAG's performance under two extreme scenarios: relying solely on internal knowledge (retrieve only) and using retrieval in each subquery (parametric only). As shown in Figure 4, parametric only yields poor performance, while retrieve only achieves relatively higher accuracy but incurs substantial retrieval costs. DeepRAG achieves superior performance by adaptively selecting between internal and external knowledge sources.

为深入理解 原子决策 的有效性，我们在两种极端场景下评估 DeepRAG：仅依赖内部知识（原文此处标作 retrieve only），以及对每个子查询都进行检索（原文此处标作 parametric only）。 如图 4所示，parametric only 的性能较差，而 retrieve only 虽能取得相对较高的准确率，却产生大量检索成本。 DeepRAG 通过在内部与外部知识源之间自适应选择，取得了更好的性能。

每个子查询中 WH 词、名词、动词及连词（and/or）的平均数量。

图8：每个子查询中 WH 词、名词、动词及连词（and/or）的平均数量。

Moreover, DeepRAG outperforms the retrieve only approach because retrieval can hinder model performance due to long context or irrelevant knowledge in certain scenarios.

此外，DeepRAG 优于 retrieve only 方法，是因为在某些场景中，检索带来的长上下文或无关知识可能损害模型性能。

5.6. Ablation Study

In this section, we conducted experiments to validate the effectiveness of DeepRAG's data construction and training process.

本节通过实验验证 DeepRAG 数据构建与训练过程的有效性。

For Imitation Learning, we compare our default strategy of selecting paths with minimal retrieval cost against two alternative approaches: maximum retrieval cost (most) and random path selection (random). As shown in Figure 5, DeepRAG-Imi achieves lower retrieval costs and higher average performance compared to both the most and random methods.

对于模仿学习，我们将默认的最小检索成本路径选择策略，与两种替代方法进行比较：选择检索成本最大的路径（most）和随机选择路径（random）。 如图 5所示，与 most 和 random 两种方法相比，DeepRAG-Imi 的检索成本更低、平均性能更高。

For Chain of Calibration, we compare our default approach of constructing preferences based on nodes from optimal paths against two alternatives: constructing pairs for all nodes and constructing sentence-level partial order pairs based on retrieval efficiency. As shown in Figure 6, DeepRAG achieves lower retrieval costs while maintaining higher average performance. In contrast, the sentence-level partial order pairs learned incorrect preferences, resulting in over-reliance on internal knowledge and consequently leading to both low retrieval costs and poor performance.

对于校准链，我们将默认的基于最优路径节点构建偏好的方法，与两种替代方案进行比较：为所有节点构建偏好对，以及基于检索效率构建句子级偏序对。 如图 6所示，DeepRAG 在保持更高平均性能的同时，取得了更低的检索成本。 相比之下，句子级偏序对学到了错误的偏好，导致模型过度依赖内部知识，因此检索成本和性能都较低。

6. Conclusion

In this paper, we present DeepRAG to model retrieval-augmented reasoning as a Markov Decision Process, enabling strategic and adaptive retrieval by decomposing queries into subqueries and retrieval on demand. Specifically, we develop a binary tree search method to synthesize data for imitation learning and further chain of calibration to train the model in an end-to-end manner. Experiments across various QA tasks show that DeepRAG improves retrieval efficiency while improving answer accuracy by 26.4%, demonstrating its effectiveness in optimizing retrieval-augmented reasoning.

本文提出 DeepRAG，将检索增强推理建模为马尔可夫决策过程，通过将查询分解为子查询并按需检索，实现具有策略性和自适应性的检索。 具体而言，我们开发了二叉树搜索方法来合成模仿学习数据，并进一步通过校准链，端到端地训练模型。 不同问答任务上的实验表明，DeepRAG 在提高检索效率的同时，将答案准确率提升了 26.4%，证明了它在优化检索增强推理方面的有效性。

Limitations

There are several limitations of our current DeepRAG framework, which we plan to address in the future. Firstly, we construct datasets based on the final answer accuracy using exact match score. In the future, we will expand our work to more domains like multi-turn dialogue with richer metrics like perplexity. Secondly, despite our method showing strong generalization across multi-hop factual QA, time-sensitive QA, and heterogeneous knowledge base QA, it lacks integration with external resources such as knowledge graphs and tools. We will expand our work to domains requiring diverse external information integration, including retrieved data, knowledge graph data, and tool output.

当前 DeepRAG 框架仍存在若干局限，我们计划在未来加以解决。 首先，我们依据最终答案的准确性，使用精确匹配分数来构建数据集。 未来，我们将把工作扩展到多轮对话等更多领域，并采用困惑度等更丰富的指标。 其次，尽管我们的方法在多跳事实问答、时间敏感问答和异构知识库问答中表现出较强泛化能力，但尚未与知识图谱和工具等外部资源整合。 我们将把工作扩展到需要融合多种外部信息的领域，包括检索数据、知识图谱数据和工具输出。