---

TaskCraft: Automated Generation of Agentic Tasks

AgentICLR 2026CCF-AOPPO

Shi D, Cao J, Chen Q, et al. TaskCraft: Automated Generation of Agentic Tasks[EB/OL]. arXiv:2506.10055, 2025.

https://openreview.net/forum?id=UJFCyrYM1V

https://github.com/OPPO-PersonalAI/TaskCraft

---

TaskCraft：智能体任务的自动生成

Abstract

Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce TaskCraft, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.

智能体任务要求自主地进行多步问题求解、使用工具并开展自适应推理，正在 NLP 与 AI 的发展中占据愈发核心的位置。 然而，现有指令数据缺少工具交互，而当前智能体基准依赖成本高昂的人工标注，限制了其可扩展性。 我们提出 TaskCraft，这是一套自动化工作流，用于生成难度可扩展、支持多工具且可验证的智能体任务及其执行轨迹。 TaskCraft 通过基于深度和基于宽度的扩展来拓展原子任务，从而构造结构与层级都更复杂的挑战。 实验结果表明，这些任务能够改进生成工作流中的提示优化，并增强智能体基础模型的监督微调。 我们发布一个包含约 36,000 个不同难度任务的大规模合成数据集，以支持未来关于智能体调优与评测的研究。

1. Introduction

Agentic tasks—autonomous, multi-step problem-solving requiring tool use and adaptive reasoning—are increasingly pivotal in AI and NLP. Advances in language agents have shifted AI from passive assistance to proactive agency, enabling complex workflow execution. This is exemplified by systems combining reasoning frameworks like ReAct with dynamic orchestration, where solution trajectories critically improve inference quality. However, the inherent complexity of such tasks challenges conventional annotation paradigms, necessitating novel approaches to model training and evaluation.

智能体任务是需要工具使用和自适应推理的自主多步问题求解，在 AI 与 NLP 中日益重要。 语言智能体的进展推动 AI 从被动辅助转向主动行动，使复杂工作流的执行成为可能。 将 ReAct 等推理框架与动态编排相结合的系统体现了这一变化，其中求解轨迹对提升推理质量至关重要。 然而，这类任务固有的复杂性对传统标注范式构成挑战，因此需要新的模型训练与评测方法。

To assess advanced agent capabilities, benchmarks such as GAIA, BrowseComp, and Humanity’s Last Exam (HLE) have been introduced. GAIA evaluates reasoning, tool use, and web browsing through 466 real-world questions. BrowseComp comprises 1,266 tasks that test an agent’s ability to retrieve and integrate complex online information. HLE includes 2,500 multi-modal questions across over 100 disciplines to measure advanced reasoning and domain knowledge. While these datasets have significantly contributed to agent evaluation, they suffer from scalability limitations due to the labor-intensive nature of data annotation. For example, creating HLE required 1,000 experts to label just 2,500 data points, hindering its ability to scale.

为了评估高级智能体能力，研究者提出了 GAIA、BrowseComp 和 Humanity’s Last Exam（HLE）等基准。 GAIA 通过 466 个现实世界问题评估推理、工具使用和网页浏览能力。 BrowseComp 包含 1,266 个任务，用于检验智能体检索并整合复杂在线信息的能力。 HLE 包含来自 100 多个学科的 2,500 个多模态问题，用于衡量高级推理与领域知识。 尽管这些数据集为智能体评测作出了重要贡献，但劳动密集型的数据标注限制了其可扩展性。 例如，构建 HLE 需要 1,000 名专家，却只标注了 2,500 个数据点，因而难以进一步扩展。

Prior work has explored the automatic generation of instruction-following data using large language models to alleviate the scalability issues of human-annotated datasets. A representative example is the Self-Instruct framework, which demonstrated that LLMs can generate high-quality, diverse instruction data for multi-turn dialogues. This approach has proven effective for supervised fine-tuning (SFT). However, these methods are primarily designed for static instruction-following scenarios and fall short in modeling agentic tasks, which require interaction with external tools and environments. Consequently, such data is insufficient for training or evaluating agents that operate in dynamic, real-world settings.

先前工作探索了使用大语言模型自动生成指令遵循数据，以缓解人工标注数据集的扩展问题。 Self-Instruct 是一个代表性例子，它表明 LLM 能够为多轮对话生成高质量且多样的指令数据。 这种方法已被证明对监督微调（SFT）有效。 然而，这些方法主要面向静态指令遵循场景，难以建模需要与外部工具和环境交互的智能体任务。 因此，这类数据不足以训练或评估在动态现实环境中运行的智能体。

In this work, we introduce TaskCraft, an agentic workflow for the automated generation of agentic tasks. Our approach provides the following advantages:

• Scalability. The workflow supports adaptive difficulty, seamless multi-tool integration, and the generation of tasks beyond the capabilities of the task-generation agent, along with their corresponding trajectories.
• Efficient Verification. During each task extension, only incremental components undergo agentic validation, eliminating the need for full verification of the extended task.

在本工作中，我们提出 TaskCraft，这是一套用于自动生成智能体任务的智能体工作流。 我们的方法具有以下优势：

• 可扩展性。 该工作流支持难度自适应、无缝的多工具集成，以及生成超出任务生成智能体能力的任务及其对应轨迹。
• 高效验证。 每次扩展任务时，只有新增部分需要经过智能体验证，无须对扩展后的整个任务进行完整验证。

The core approach involves initially generating multiple atomic tasks, each solvable with a single target tool invocation, and then expanding them using depth-based and width-based extension. For depth-based task extension, we iteratively transform specific textual elements of the original task (such as key terms) into a new atomic task to support progressive resolution. In contrast, the width-based extension formulates tasks that require resolving multiple sub-problems by integrating distinct problem instances.

核心方法首先生成多个原子任务，每个任务都能通过一次目标工具调用解决，再使用基于深度和基于宽度的扩展来拓展这些任务。 在基于深度的任务扩展中，我们迭代地把原任务中的特定文本元素（如关键词）转化为新的原子任务，以支持逐步求解。 相比之下，基于宽度的扩展通过整合不同的问题实例，构造需要解决多个子问题的任务。

To ensure high-quality agentic tasks, we employ a rejection sampling strategy during verification. For atomic tasks, we include cases where an agent using external tools can solve the task while an LLM cannot, ensuring that atomic tasks genuinely necessitate tool usage. For extension tasks, we leverage linguistic analysis with LLMs, enabling rapid validation and facilitating the creation of challenges beyond existing agent capabilities. This approach enhances efficiency and broadens problem-solving potential.

为了确保智能体任务的高质量，我们在验证期间采用拒绝采样策略。 对于原子任务，我们保留使用外部工具的智能体能够解决、但 LLM 无法解决的样例，从而确保原子任务确实需要使用工具。 对于扩展任务，我们利用 LLM 进行语言分析，以实现快速验证，并促进构造超出现有智能体能力的挑战。 这种方法提高了效率，也拓宽了解题空间。

The controlled generation process ensures inherent access to ground-truth execution trajectories, enabling precise interpretability, reproducibility, and verifiability—critical for agent evaluation and reinforcement learning. To further validate task effectiveness, we implement a self-evolving prompt optimization strategy inspired by bootstrap few-shot learning. This iterative refinement improves rejection sampling pass rates while minimizing generation time. Additionally, we leverage the generated task trajectories to train an agent foundation model. Experimental results show that an independent LLM, trained on these trajectories, effectively plans and invokes tools, yielding performance gains on HotpotQA, Musique, and Bamboogle.

受控生成过程天然提供真实执行轨迹，从而带来精确的可解释性、可复现性与可验证性，而这些特性对智能体评测和强化学习至关重要。 为了进一步验证任务的有效性，我们实现了一种受自举少样本学习启发的自演化提示优化策略。 这种迭代改进在缩短生成时间的同时，提高了拒绝采样的通过率。 此外，我们利用生成的任务轨迹训练智能体基础模型。 实验结果表明，在这些轨迹上训练的独立 LLM 能够有效规划并调用工具，在 HotpotQA、Musique 和 Bamboogle 上取得性能提升。

Based on this method, we generated a task dataset comprising approximately 36,000 tasks of varying difficulty, each requiring different tools for resolution, including search, web browsing, PDF reading, and image understanding.

基于该方法，我们生成了一个包含约 36,000 个不同难度任务的数据集，每个任务都需要使用不同工具求解，包括搜索、网页浏览、PDF 阅读和图像理解。

Our key contributions are as follows:

• We introduce an automated agentic task generation workflow capable of producing scalable difficulty, efficient verification, and multi-tool supported tasks, along with their corresponding execution trajectories.
• We empirically evaluate task effectiveness through prompt learning, which facilitates the self-evolution of our workflow and holds potential for optimizing existing agent workflows. Additionally, SFT is applied to an agent foundation model, enabling it to substitute agent workflows where applicable.
• We release a synthetic dataset comprising about 32k agentic tasks of varying difficulty levels, complete with their execution trajectories, to facilitate further research.

我们的主要贡献如下：

• 我们提出一套自动化智能体任务生成工作流，能够生成难度可扩展、可高效验证且支持多工具的任务，以及相应的执行轨迹。
• 我们通过提示学习实证评估任务有效性，这促进了工作流的自演化，并有望用于优化现有智能体工作流。此外，我们将 SFT 应用于智能体基础模型，使其在适用场景中能够替代智能体工作流。
• 我们发布一个包含约 32k 个不同难度智能体任务及其完整执行轨迹的合成数据集，以促进后续研究。

2. Notations and Preliminary

Tool-Assisted Task Execution. As Figure 1 shown, given a task $q$, the agent extracts the input index $i_T$ (e.g., document name, webpage title) for invoking a target tool $T$. We focus solely on steps that yield a valid tool context, omitting unrelated processes such as file location or search for simplicity. Executing tool $T$ with $i_T$ retrieves the associated context $C$. The LLM implicitly deduces the relationship $R$ between $C$ and the expected outcome, producing the final result $a$.

工具辅助的任务执行。 如图1所示，给定任务 $q$，智能体提取输入索引 $i_T$（例如文档名、网页标题），以调用目标工具 $T$。 为简化起见，我们只关注能够产生有效工具上下文的步骤，并省略文件定位或搜索等无关过程。 使用 $i_T$ 执行工具 $T$ 会检索到相关上下文 $C$。 LLM 隐式推断 $C$ 与预期结果之间的关系 $R$，并生成最终结果 $a$。

图1：单次工具调用的执行流程。智能体提取用于调用工具 T 的输入索引 i_T（例如文档名、网页标题），并仅关注能够产生有效工具上下文的步骤。使用 i_T 执行 T 会检索上下文 C，使 LLM 能够推断关系 R 并生成最终结果 a。

Atomic Task. An atomic task is resolved with a single target tool invocation. To simplify, we disregard search and file system operations, assuming a detailed input index $i_T$ enables retrieval through finite navigation.

原子任务。 原子任务通过一次目标工具调用即可解决。 为简化起见，我们忽略搜索和文件系统操作，并假定详细的输入索引 $i_T$ 能够通过有限次导航实现检索。

Given an answer $a$, the most direct approach to construct an atomic task involves prompting an LLM to generate the corresponding question. However, questions produced in this manner often suffer from low tool invocation rates, unpredictable difficulty levels, unregulated tool requirements, and inconsistent verification complexity (see Section 4.5 for more details). To mitigate these issues, we assume an ideal search engine capable of retrieving precise data based on $i_T$ (e.g., paper titles, image paths, music names, etc.). Under this assumption, we can construct a task question $q=f(i_T,R)\stackrel{}{\to }a$, where $f$ represents a sampling function that enables the LLM to generate the corresponding natural language representation of the question $q$ based on the provided information.

给定答案 $a$，构造原子任务最直接的方法是提示 LLM 生成相应问题。 然而，以这种方式生成的问题往往存在工具调用率低、难度不可预测、工具需求不受约束以及验证复杂度不一致等问题（详见第 4.5 节）。 为了缓解这些问题，我们假设存在一个理想搜索引擎，能够根据 $i_T$（例如论文标题、图像路径、音乐名称等）检索精确数据。 在这一假设下，我们可以构造任务问题 $q=f(i_T,R)\stackrel{}{\to }a$，其中 $f$ 表示一个采样函数，使 LLM 能够根据给定信息生成问题 $q$ 对应的自然语言表述。

3. Automated Task Generation Workflow

图2：原子任务生成：从无标注语料中提取 i_T，并通过工具执行得到文本内容 C。LLM 从 C 中识别候选答案 a，推断其关系 R，并以 i_T 和 R 为条件构造问题 q。

3.1 Atomic Task Generation

As Figure 2 shown, we begin by compiling a corpus of unlabeled data aligned with the tool's input requirements. From this corpus, we extract $i_T$ and derive textual content $C$ via tool execution. For example, browsing, PDF, and image comprehension tools yield webpage titles, PDF names, and image paths, from which we extract textual content $C$ for answer sampling. We prompt an LLM to identify key candidate answers $a$ from $C$ and infer their relationship $R$ with $C$, ultimately constructing question $q$ conditioned on $i_T$ and $R$.

如图2所示，我们首先汇集与工具输入要求相匹配的无标注数据语料。 我们从该语料中提取 $i_T$，并通过工具执行得到文本内容 $C$。 例如，浏览、PDF 和图像理解工具会产生网页标题、PDF 名称和图像路径，我们从中提取文本内容 $C$ 以采样答案。 我们提示 LLM 从 $C$ 中识别关键候选答案 $a$，并推断它们与 $C$ 的关系 $R$，最终以 $i_T$ 和 $R$ 为条件构造问题 $q$。

3.2 Task Extension

In order to increase task difficulty in a scalable way, we adopted two extended task strategies: the depth-based extension and the width-based extension.

为了以可扩展方式提高任务难度，我们采用了两种任务扩展策略：基于深度的扩展和基于宽度的扩展。

Depth-based extension. We aim to construct tasks requiring multiple sequential tool executions, where each step depends on the output of the previous one. To achieve this, a new subproblem must be derived from a known problem $q^n$. The tool input index $i_T$ at each stage exhibits strong extensibility due to (1) its frequent association with proper nouns, which are less likely to be memorized by LLMs, and (2) its natural suitability for recursive definition. Specifically, a single atomic task follows the formulation:

基于深度的扩展。 我们旨在构造需要连续执行多次工具的任务，其中每一步都依赖前一步的输出。 为此，必须从已知问题 $q^n$ 派生一个新的子问题。 每一阶段的工具输入索引 $i_T$ 都具有很强的可扩展性，因为：（1）它经常与专有名词相关，而 LLM 不太可能记住这些专有名词；（2）它天然适合递归定义。 具体而言，单个原子任务遵循以下形式：

$$q^n=f(i_T^n,R^n)\stackrel{}{\to }a.$$

图3：基于深度的扩展。一个 1-hop 任务 q^1 被递归扩展为 2-hop 任务 q^2。搜索智能体通过提取超集候选 C^2 得到新的工具输入索引 i_T^2，LLM 随后分析并确定 i_T^2 及其与 i_T^1 的关系 R^2。验证后，改进的问题 q^2 将 q̂^2 与历史关系 R^1 相结合。

To extend a n-hot task $q^n$ into a (n+1)-hop dependency task $q^{n+1}$, we can define the recursive formulation:

为了把一个 n-hot 任务 $q^n$ 扩展为一个 (n+1)-hop 依赖任务 $q^{n+1}$，我们可以定义如下递归形式：

$$q^{n+1}=f({\hat{q}}^{n+1},R^n)\stackrel{}{\to }a.$$

where we ensure that

其中，我们确保

$${\hat{q}}^{n+1}=f(i_T^{n+1},R^{n+1})\stackrel{}{\to }{i}_T^n.$$

Here, $i_T^{n+1}$ denotes a new tool input index derived from $i_T^n$ through reversible operations (e.g., retrieving lyrics from a song name or vice versa). To obtain $i_T^{n+1}$ and its corresponding relationship $R^{n+1}$, we employ a search agent that retrieves supersets of $i_T^n$ to mitigate cyclic generation risks. Specifically, the agent extracts textual content $C^{n+1}$ as superset candidates, expanding contextual coverage. An LLM then analyzes $C^{n+1}$ to derive the superset index $i_T^{n+1}$ and its relationship $R^{n+1}$ with $i_T^n$. This process ensures progressive context expansion and effective information association. The resulting $i_T^{n+1}$ and $R^{n+1}$ are synthesized into an intermediate question candidate ${\hat{q}}^{n+1}$, which undergoes rigorous verification. Upon verification, the system generates the refined question $q^{n+1}$ by integrating ${\hat{q}}^{n+1}$ with all historical relationships $\{R^1,R^2,...,R^n\}$.

这里，$i_T^{n+1}$ 表示通过可逆操作从 $i_T^n$ 派生的新工具输入索引（例如，根据歌曲名检索歌词，或反向操作）。 为了获得 $i_T^{n+1}$ 及其对应关系 $R^{n+1}$，我们使用搜索智能体检索 $i_T^n$ 的超集，以减轻循环生成风险。 具体而言，智能体提取文本内容 $C^{n+1}$ 作为超集候选，从而扩大上下文覆盖范围。 随后，LLM 分析 $C^{n+1}$，得到超集索引 $i_T^{n+1}$ 及其与 $i_T^n$ 的关系 $R^{n+1}$。 这一过程确保上下文逐步扩展并有效关联信息。 所得 $i_T^{n+1}$ 和 $R^{n+1}$ 被合成为中间问题候选 ${\hat{q}}^{n+1}$，并接受严格验证。 验证通过后，系统将 ${\hat{q}}^{n+1}$ 与所有历史关系 $\{R^1,R^2,...,R^n\}$ 相结合，生成改进后的问题 $q^{n+1}$。

Width-based extension. The goal of the width-based extension is to generate a new task that needs to be decoupled into multiple subtasks to be completed. For simplicity, for two subtasks $q_1\stackrel{}{\to }{a}_1$ and $q_2\stackrel{}{\to }{a}_2$, the combined task $q_{width}$ can be represented as

基于宽度的扩展。 基于宽度的扩展旨在生成一个必须拆分为多个子任务才能完成的新任务。 为简化起见，对于两个子任务 $q_1\stackrel{}{\to }{a}_1$ 和 $q_2\stackrel{}{\to }{a}_2$，组合任务 $q_{width}$ 可以表示为

$$(q_{width}=q_1+q_2)\stackrel{}{\to }{a}_1+a_2,$$

where the $+$ indicates using LLM to merge and rephrase two question strings.

其中，$+$ 表示使用 LLM 合并并改写两个问题字符串。

图4：基于宽度的扩展。通过合并两个子任务 q_1 和 q_2 形成新任务，得到 q_width = q_1 + q_2，其中 + 表示基于 LLM 的改写。

Trajectory generation. Two strategies exist for generating execution trajectories in this task: (1) For simple tasks, such as atomic tasks, existing agents can directly infer and capture the trajectory, including tool selection, parameters, return results, and plans. (2) For complex tasks, such as depth-wise extension tasks, the trajectory is recorded while iteratively expanding and validating new atomic tasks. At each step, the LLM refines the plan or reasoning based on generated intermediate questions.

轨迹生成。 该任务有两种执行轨迹生成策略：（1）对于原子任务等简单任务，现有智能体可以直接推断并记录轨迹，包括工具选择、参数、返回结果和规划。 （2）对于基于深度的扩展任务等复杂任务，系统在迭代扩展并验证新原子任务的同时记录轨迹。 在每一步，LLM 都根据生成的中间问题改进规划或推理。

3.3 Task Verification

Under this generation workflow, the verification of generated tasks can be easily performed in two distinct phases:

在该生成工作流下，可以通过两个不同阶段轻松验证生成的任务：

Atomic task verification. An atomic task is defined as a simple agent task solvable via a single tool call. During verification, we relax this definition slightly: for each candidate task, we evaluate the task agent’s output within a limited number of tool-use steps (e.g., three) and compare it with an infer-LLM separately. A judge-LLM verifies whether only the agent’s output contains the golden answer, retaining only validated tasks. See the appendix for more details.

原子任务验证。 原子任务被定义为能够通过一次工具调用解决的简单智能体任务。 验证时，我们略微放宽这一定义：对于每个候选任务，我们在有限的工具使用步数（例如三步）内评估任务智能体的输出，并单独与推断 LLM 比较。 评判 LLM 验证是否只有智能体输出包含标准答案，并仅保留通过验证的任务。 更多细节见附录。

Task extension verification. This process is conducted purely through linguistic analysis without agent involvement. During depth-wise extension, we first employ a judge-LLM to validate: (1) whether the obtained $i_T^{n+1}$ and its relation $R^{n+1}$ constitute a proper superset of $i_T^n$ with logically sound relationships, and (2) whether the final input index $i_T^n$ in $q^n$ is appropriately replaced by ${\hat{q}}^{n+1}$ in the expanded task $q^{n+1}$. Furthermore, an infer-LLM derives the merged task, while the judge-LLM filters out tasks where the correct result is easily inferred, preventing information leakage that could render the problem trivially solvable after merging. See the appendix for more details.

任务扩展验证。 这一过程完全通过语言分析完成，不需要智能体参与。 在基于深度的扩展期间，我们首先使用评判 LLM 验证：（1）所得 $i_T^{n+1}$ 及其关系 $R^{n+1}$ 是否构成 $i_T^n$ 的适当超集，且关系在逻辑上成立；（2）在扩展任务 $q^{n+1}$ 中，$q^n$ 的最终输入索引 $i_T^n$ 是否被 ${\hat{q}}^{n+1}$ 正确替换。 此外，推断 LLM 得到合并后的任务，而评判 LLM 会过滤正确结果很容易被推断出的任务，以避免信息泄漏导致合并后的问题过于简单。 更多细节见附录。

This framework ensures efficiency by applying agent reasoning only in atomic task verification at creation, while relying on LLM-based verification elsewhere for faster execution. It also enables complex task generation beyond agent capabilities, with reverse reasoning providing supervisory signals to enhance agent learning or reinforcement learning.

该框架只在创建原子任务时使用智能体推理进行验证，其他环节则依靠基于 LLM 的验证以加快执行，从而保证效率。 它还支持生成超出智能体能力的复杂任务，并通过反向推理提供监督信号，以增强智能体学习或强化学习。

4. Experiments

4.1 Corpus Construction

图5：语料来源分布。网页、PDF 和图像经过处理后用于构造工具特定任务。

We collect seed documents across multiple modalities to generate tool-specific atomic tasks, extracting key insights to ensure task relevance. For instance, our PDF processor constructs atomic tasks by combining document titles with core findings, thereby enhancing the necessity for agent-based PDF tool invocation. To support atomic task generation, we constructed a dataset comprising webpages, PDF files, and images. Webpage data constitutes the largest proportion (75%), sourced from up-to-date news across multiple domains. Image data accounts for 15%, primarily derived from financial reports and research papers, with filtering to retain images containing information beyond text. PDF data makes up 10%, originating from English financial documents and academic publications.

我们收集多种模态的种子文档来生成工具特定的原子任务，并提取关键信息以确保任务相关性。 例如，我们的 PDF 处理器将文档标题与核心发现相结合来构造原子任务，从而增强调用智能体 PDF 工具的必要性。 为了支持原子任务生成，我们构建了一个由网页、PDF 文件和图像组成的数据集。 网页数据占比最大（75%），来源是多个领域的最新新闻。 图像数据占 15%，主要来自财务报告和研究论文，并经过筛选以保留包含文本以外信息的图像。 PDF 数据占 10%，来自英文财务文档和学术出版物。

4.2 Synthetic Tasks Analysis

Agent reasoning analysis. To practically assess task difficulty, we sample 1,000 tasks and deploy both Smolagents and its enhanced variant, Smolagents+ (see the appendix for more details), for execution and validation. While both agents performed identical tasks, Smolagents+ incorporated advanced tool capabilities for refined analysis.

智能体推理分析。 为了在实践中评估任务难度，我们采样 1,000 个任务，并部署 Smolagents 及其增强变体 Smolagents+（更多细节见附录）进行执行和验证。 两个智能体执行相同任务，但 Smolagents+ 集成了高级工具能力，以开展更精细的分析。

图6：分数分布比较。

Responses were evaluated by comparing the agents' outputs to the golden answer, following a three-point scoring scheme: 2 for fully correct responses, 1 for answers that included the golden answer but contained additional information, and 0 for incorrect responses.

我们通过比较智能体输出与标准答案来评估回答，并采用三分制评分方案：完全正确的回答得 2 分，包含标准答案但带有额外信息的回答得 1 分，错误回答得 0 分。

In Figure 6, task failure rates increase from web pages to PDFs and then to images within PDFs, indicating that multi-hop web search tasks are more manageable for agents, while complex comprehension challenges, such as PDF extraction and image interpretation, remain difficult. Additionally, these results demonstrate that our generated tasks span varying difficulty levels, including those that pose significant challenges for current agent capabilities.

在图6中，任务失败率从网页到 PDF、再到 PDF 内图像依次上升，这表明多跳网页搜索任务对智能体而言更易处理，而 PDF 提取和图像解释等复杂理解挑战仍然困难。 此外，这些结果表明，我们生成的任务覆盖了不同难度，其中包括对当前智能体能力构成显著挑战的任务。

Comparison with the GAIA dataset. Table 1 presents the accuracy comparison of Smolagent on the GAIA dataset and our generated dataset. The results indicate that tasks derived from different tool corpora align with GAIA’s varying difficulty levels, with image understanding tasks posing the greatest challenge and achieving accuracy comparable to LEVEL3 data.

与 GAIA 数据集比较。 表1给出了 Smolagent 在 GAIA 数据集和我们生成的数据集上的准确率比较。 结果表明，来自不同工具语料的任务与 GAIA 的不同难度等级相对应，其中图像理解任务最具挑战性，其准确率与 LEVEL3 数据相当。

表1：Smolagents 在 GAIA 数据集和我们的合成任务上的准确率比较。

GAIA	Level1	Level2	Level3	Avg.
	54.71	43.02	26.92	44.20
Synthetic Task	PDF	html	Image	Avg.
	54.4	50.7	22.1	42.4

Unlike GAIA, which requires extensive human annotation, our approach automates task generation, eliminating the need for labor-intensive data labeling while maintaining scalability and adaptability for agent self-evolution and optimization.

GAIA 需要大量人工标注，而我们的方法实现了任务生成自动化，在保持智能体自演化与优化所需的可扩展性和适应性的同时，免除了劳动密集型的数据标注。

图7：需要多次工具调用才能完成的生成案例。

4.3 Enhancing Task Generation Efficiency via Prompt Learning

We employ rejection sampling in both atomic task generation and task extension. To reduce the rejection rate and enhance sampling efficiency, several key challenges must be addressed:

• Efficiently extract candidate answers from the corpus to support atomic task formation and minimize rejections (Section 3.1).
• Guide the agent to find an input index $i_T^{n+1}$, ensuring coherent depth-wise extension.
• Prompt the LLM in depth-wise extension to articulate the relationship $R^{n+1}$ between the previous input index $i_T^n$ and observed content $C^{n+1}$, refining problem construction and mitigating incoherence-related rejections.
• Integrate tasks to ensure precise substitution, i.e., $q^{n+1}=f({\hat{q}}^{n+1},R^n)$, and clarity while maintaining logical coherence.

我们在原子任务生成和任务扩展中都采用拒绝采样。 为了降低拒绝率并提高采样效率，必须解决几个关键挑战：

• 从语料中高效提取候选答案，以支持原子任务构造并尽量减少拒绝（第 3.1 节）。
• 引导智能体找到输入索引 $i_T^{n+1}$，确保基于深度的扩展保持连贯。
• 在基于深度的扩展中提示 LLM 阐明前一输入索引 $i_T^n$ 与观测内容 $C^{n+1}$ 之间的关系 $R^{n+1}$，从而改进问题构造并减少因不连贯造成的拒绝。
• 整合任务，在保持逻辑连贯的同时确保替换准确，即 $q^{n+1}=f({\hat{q}}^{n+1},R^n)$，并保证表述清晰。

Evaluation. We assess atomic task generation and task extension separately. For atomic task generation, we evaluate three key metrics: (1) pass rate, representing the proportion of successfully validated atomic tasks relative to candidate tasks. (2) task density, quantifying the average number of validated atomic tasks per document. (3) sampling time, measuring the time required for processing each document. For task extension, we evaluate three key metrics: (1) pass rate, the proportion of successful extensions across $n_k$ attempts (set to 6 in our experiment). (2) sampling time, measuring the time required for extending each task.

评估。 我们分别评估原子任务生成和任务扩展。 对于原子任务生成，我们评估三个关键指标：（1）通过率，即成功验证的原子任务占候选任务的比例。 （2）任务密度，即每份文档中通过验证的原子任务平均数量。 （3）采样时间，即处理每份文档所需的时间。 对于任务扩展，我们评估三个关键指标： （1）通过率，即 $n_k$ 次尝试中成功扩展的比例（实验中设为 6）。 （2）采样时间，即扩展每个任务所需的时间。

Prompt Learning. Intuitively, providing the LLM with effective exemplars can further enhance its ability to identify intermediate objectives. To this end, we employ bootstrap few-shot learning to systematically optimize the four prompts corresponding to the aforementioned challenges, thereby facilitating the generated workflow. For atomic task generation, each prompt is optimized by appending 20 randomly sampled examples. Multiple prompt configurations are then generated by varying these samples, followed by an iterative evaluation process where pass rates determine the optimal selection of inserted examples. For task extension, we focus on depth-wise extension and adopt a similar strategy to optimize the prompts using 10 randomly sampled examples. These prompts are refined to maximize the number of hops.

提示学习。 直观而言，为 LLM 提供有效示例可以进一步增强其识别中间目标的能力。 为此，我们采用自举少样本学习，系统地优化与上述挑战相对应的四个提示，从而改进生成工作流。 对于原子任务生成，每个提示都通过附加 20 个随机采样示例来优化。 随后，我们通过改变这些样本生成多种提示配置，并进行迭代评估，以通过率决定插入示例的最佳选择。 对于任务扩展，我们聚焦基于深度的扩展，并采用类似策略，使用 10 个随机采样示例优化提示。 这些提示会被不断改进，以最大化跳数。

表2：生成的任务数据在提示学习中的有效性，以及六次扩展尝试下基于深度扩展的结果。

Method	Pass rate	Time
Atomic Task	54.9%	29.1s
+ Optimization	68.1%	23.5s
Depth-wise@6	41.0%	31.5s
+ Optimization	51.2%	30.2s

Results. Table 2 examines atomic task generation and depth-wise task extension before and after prompt learning, highlighting the role of generated task data in enabling self-evolution within both workflows. For atomic task generation, the data improves efficiency by reducing generation time by $19.2\mathrm{\%}$ (29.1 to 23.5 seconds) and increasing pass rate from $54.9\mathrm{\%}$ to $68.1\mathrm{\%}$. Similarly, depth-wise extension benefits from the data, with pass rate rising by $10.2\mathrm{\%}$ ($41.0\mathrm{\%}$ to $51.2\mathrm{\%}$) across six extension attempts, and generation time decreasing by 1.3 seconds (31.5 to 30.2 seconds). These results validate the effectiveness of generated task data in enhancing sampling efficiency and supporting workflow adaptation. The optimized prompts are presented in the appendix.

结果。 表2考察了提示学习前后的原子任务生成和基于深度的任务扩展，突出了生成任务数据在两个工作流中实现自演化的作用。 对于原子任务生成，这些数据把生成时间缩短 $19.2\mathrm{\%}$（从 29.1 秒降至 23.5 秒），并将通过率从 $54.9\mathrm{\%}$ 提高到 $68.1\mathrm{\%}$，从而提升效率。 类似地，基于深度的扩展也受益于这些数据：在六次扩展尝试中，通过率提高 $10.2\mathrm{\%}$（从 $41.0\mathrm{\%}$ 提高到 $51.2\mathrm{\%}$），生成时间则缩短 1.3 秒（从 31.5 秒降至 30.2 秒）。 这些结果验证了生成任务数据在提高采样效率和支持工作流适应方面的有效性。 优化后的提示见附录。

4.4 Fine-Tuning Agent Models Using Synthetic Trajectory

To validate the effectiveness of our synthetic multi-hop data method, we apply supervised fine-tuning (SFT) and reinforcement learning (RL) using the generated trajectory, refining an agent foundation model—an LLM with tool-integrated reasoning.

为了验证合成多跳数据方法的有效性，我们使用生成的轨迹进行监督微调（SFT）和强化学习（RL），以改进智能体基础模型，即集成工具推理的 LLM。

Evaluation. We evaluate our models on three multi-hop question answering benchmark datasets, as follows: HotpotQA, Musique, and Bamboogle. These datasets encompass a diverse range of search with reasoning challenges, enabling a comprehensive evaluation.

评估。 我们在三个多跳问答基准数据集上评估模型：HotpotQA、Musique 和 Bamboogle。 这些数据集涵盖多种搜索与推理挑战，能够实现全面评估。

Baselines. We conduct a comprehensive evaluation by comparing various baseline models before and after SFT with generated tasks to assess performance improvements: (1) Base workflow: We implement agent workflows (Search-R1 without training) across different LLM models. (2) Search-R1: An agentic workflow leveraging reinforcement learning for LLM model optimization.

基线。 为了评估性能提升，我们比较多种基线模型在使用生成任务进行 SFT 前后的表现，从而开展全面评估：（1）基础工作流：我们在不同 LLM 模型上实现智能体工作流（未经训练的 Search-R1）。 （2）Search-R1：一种利用强化学习优化 LLM 模型的智能体工作流。

Implementation setup. We evaluate two model variants: Qwen2.5-3B-Base and Qwen2.5-3B-Instruct. To facilitate multi-hop reasoning, we synthesize 3,202 multi-hop tasks and their trajectories for SFT. Following the Chain-of-Action framework, we apply content masking to search tool contexts during training. Our search method, RL training data, and reinforcement learning strategy follow the Search-R1. For further training details, refer to the appendix.

实现设置。 我们评估两个模型变体：Qwen2.5-3B-Base 和 Qwen2.5-3B-Instruct。 为了支持多跳推理，我们为 SFT 合成了 3,202 个多跳任务及其轨迹。 遵循 Chain-of-Action 框架，我们在训练期间对搜索工具上下文应用内容掩码。 我们的搜索方法、RL 训练数据和强化学习策略遵循 Search-R1。 更多训练细节见附录。

表3：两个模型在三个数据集上的性能。Avg. 表示平均值。

Method	HotpotQA	Musique	Bamboogle	Avg.
Qwen2.5-3b-Base
Base workflow	0.032	0.006	0.063	0.034
+ SFT	0.232	0.067	0.224	0.174
Search-R1	0.284	0.049	0.088	0.140
+ SFT	0.344	0.111	0.280	0.245
Qwen2.5-3b-Instruct
Base workflow	0.190	0.037	0.112	0.113
+ SFT	0.221	0.049	0.248	0.173
Search-R1	0.324	0.103	0.264	0.230
+ SFT	0.340	0.104	0.264	0.236

Results. As shown in Table 3, our method demonstrates significant performance improvements across three representative datasets and two model variants. First, our synthetic data demonstrates significant value in standalone SFT training, achieving average performance improvements of +14.0% (Qwen2.5-3B-Base) and +6.0% (Qwen2.5-3B-Instruct) over the base workflow for their respective models. These gains validate the quality and effectiveness of our synthetic data generation methodology. Second, compared to the Search-R1 baseline, the workflow with Qwen2.5-3b-Base achieves maximum gains of +19.2% on Bamboogle and +6.2% on Musique. The Qwen2.5-3B-Instruct maintains steady gains, with an average performance margin of +0.6%. The strong performance of our SFT-trained models underscores their suitability for subsequent reinforcement learning, suggesting that our synthetic data not only enhances immediate task execution but also provides a more effective initialization for RL optimization.

结果。 如表3所示，我们的方法在三个代表性数据集和两个模型变体上都取得了显著性能提升。 首先，我们的合成数据在独立 SFT 训练中体现出显著价值，相较各自模型的基础工作流，Qwen2.5-3B-Base 和 Qwen2.5-3B-Instruct 的平均性能分别提高 +14.0% 和 +6.0%。 这些增益验证了合成数据生成方法的质量与有效性。 其次，与 Search-R1 基线相比，使用 Qwen2.5-3b-Base 的工作流在 Bamboogle 上最高提升 +19.2%，在 Musique 上提升 +6.2%。 Qwen2.5-3B-Instruct 保持稳定增益，平均性能优势为 +0.6%。 SFT 训练模型的强劲表现凸显了它们对后续强化学习的适用性，这说明我们的合成数据不仅增强了即时任务执行，还为 RL 优化提供了更有效的初始化。

4.5 Effectiveness of Tool Context in Constructing Agentic Tasks

In atomic task generation, we integrate the additional input index $i_T$ along with the relational mapping $R$ between the tool context and a given answer to systematically structure tasks. To assess the efficiency of our atomic task generation approach, we perform an ablation study using an LLM to directly generate a task $q$ that requires only one external tool to obtain the answer $a$, explicitly excluding the conditions $i_T$ and $R$. Evaluation metrics include pass rate, task resolution time, average tool usage, and the variance in tool usage frequency.

在原子任务生成中，我们加入额外输入索引 $i_T$，以及工具上下文与给定答案之间的关系映射 $R$，以系统化地构造任务。 为了评估原子任务生成方法的效率，我们开展一项消融研究：使用 LLM 直接生成只需一个外部工具即可得到答案 $a$ 的任务 $q$，并明确排除条件 $i_T$ 和 $R$。 评估指标包括通过率、任务解决时间、平均工具使用次数以及工具使用频率的方差。

表4：工具上下文的有效性。

Method	Pass rate	Time	#Tool-use	σ²
LLM only	18.5%	119.7s	2.8	1.2
Ours	43.0%	86.7s	2.1	0.4

Compared to atomic tasks generated via direct prompting of GPT-4.1, our approach significantly enhances atomic task generation efficiency. Specifically, our workflow achieves a 24.5% higher pass rate (43.0% vs. 18.5%) while reducing task generation time by 28 seconds (86.7s vs. 119.7s), underscoring the limitations of vanilla LLMs in constructing agentic tasks. Furthermore, our atomic tasks exhibit greater atomicity, as evidenced by a lower average tool invocation count (2.1 vs. 2.8 per query). Task complexity also remains more stable and controllable, with a reduced variance in tool usage (0.4 vs. 1.2). These findings underscore the robustness of our workflow, validating its efficacy in structured task generation.

与直接提示 GPT-4.1 生成的原子任务相比，我们的方法显著提高了原子任务生成效率。 具体而言，我们的工作流将通过率提高了 24.5%（43.0% 对 18.5%），同时把任务生成时间缩短了 28 秒（86.7 秒对 119.7 秒），这凸显了普通 LLM 在构造智能体任务方面的局限。 此外，我们的原子任务具有更强的原子性，这体现为更低的平均工具调用次数（每个查询 2.1 次对 2.8 次）。 任务复杂度也更加稳定且可控，工具使用方差从 1.2 降至 0.4。 这些发现凸显了工作流的稳健性，并验证了其在结构化任务生成中的有效性。

5. Related Work

5.1 Instruction Data Generation

Synthetic data has emerged as a promising solution for enhancing performance and enabling new capabilities. STaR augments learning with chain-of-thought (CoT) rationales but often requires a substantial number of task queries beforehand. Methods such as Self-Instruct, Self-Chat, NuminaMath, and OpenMathInstruct-2 generate data from minimal seed examples using LLMs, yet they struggle to extend task generation for multiple tool invocations.

合成数据已成为提升性能并赋予新能力的一种前景可观的方案。 STaR 使用思维链（CoT）推理过程增强学习，但通常需要预先准备大量任务查询。 Self-Instruct、Self-Chat、NuminaMath 和 OpenMathInstruct-2 等方法利用 LLM 从少量种子示例生成数据，但难以把任务生成扩展到需要多次工具调用的场景。

WizardLM employs Evol-Instruct to incrementally enhance instruction complexity. However, it relies primarily on rule-based modifications, making its generated instructions unsuitable for agentic task scenarios.

WizardLM 使用 Evol-Instruct 逐步提高指令复杂度。 然而，它主要依赖基于规则的修改，使生成的指令不适合智能体任务场景。

MetaMath generates mathematical data by rewriting questions, but adapting agent tasks to environmental feedback presents challenges beyond simple rephrasing. WebInstruct extracts question-answer pairs from a pre-training corpus across multiple domains; however, the generated questions often fail to incorporate tool utilization in their solutions. AutoAct uses a self-planning mechanism to generate planning trajectories for QA tasks.

MetaMath 通过改写问题生成数学数据，但让智能体任务适应环境反馈所面临的挑战超出了简单改述。 WebInstruct 从跨多个领域的预训练语料库中提取问答对；然而，生成的问题通常未能在解答中纳入工具使用。 AutoAct 使用自规划机制为问答任务生成规划轨迹。

5.2 Language Agent

Existing research on agentic task execution primarily advances along two core dimensions: role specialization and functional partitioning. Role-based paradigms structure collaborative networks by dynamically allocating differentiated tools, as demonstrated by AutoGPT, AutoGen, and Camel. In contrast, functional partitioning frameworks, such as Barcelona2, Omne, and AgentIM, define distinct task execution roles, optimizing modular efficiency. These are closed-source frameworks. Smolagents combines the ReAct and CodeAct architectures to build a multi-functional agents hierarchy to perform multiple rounds of interactions and actions in code to accomplish complex tasks. Magnetic-One refines vision-language processing by decoupling perception, planning, and execution modules, improving efficiency in multimodal environments. Dynamic orchestration mechanisms address real-time task reallocation and system resilience. Trase-Agent adapts execution strategies based on real-time feedback, while TapeAgents employs asynchronous communication to enhance robustness in agent coordination. Empirical findings suggest that stabilized sub-agent interactions yield higher task success rates than complex, centralized orchestration algorithms.

现有关于智能体任务执行的研究主要沿两个核心维度推进：角色专业化与功能划分。 基于角色的范式通过动态分配差异化工具来组织协作网络，AutoGPT、AutoGen 和 Camel 都体现了这一点。 相比之下，Barcelona2、Omne 和 AgentIM 等功能划分框架定义不同的任务执行角色，以优化模块效率。 这些框架均为闭源框架。 Smolagents 结合 ReAct 与 CodeAct 架构，构建多功能智能体层级，通过多轮交互和代码操作完成复杂任务。 Magnetic-One 将感知、规划和执行模块解耦，以改进视觉语言处理，并提高多模态环境中的效率。 动态编排机制处理实时任务重分配和系统韧性问题。 Trase-Agent 根据实时反馈调整执行策略，而 TapeAgents 使用异步通信增强智能体协调的稳健性。 实证结果表明，稳定的子智能体交互比复杂的集中式编排算法带来更高的任务成功率。

To further extend agentic autonomy, AutoAgent facilitates intelligent execution and personalized agent customization without requiring manual coding. Its core components—natural language-driven coordination, customizable workflows, and self-managing file systems—streamline agent development. Hybrid architectures, such as h2oGPTe-Agent, explore multi-agent optimization strategies, achieving over 70% accuracy in code generation tasks. However, significant cross-modal processing bottlenecks remain an open challenge.

为了进一步扩展智能体自主性，AutoAgent 无须人工编码即可实现智能执行和个性化智能体定制。 它的核心组件包括自然语言驱动的协调、可定制工作流和自管理文件系统，能够简化智能体开发。 h2oGPTe-Agent 等混合架构探索多智能体优化策略，在代码生成任务上取得超过 70% 的准确率。 然而，显著的跨模态处理瓶颈仍是一个开放挑战。

6. Conclusion

We present TaskCraft, an automated workflow for scalable, multi-tool, verifiable agentic task generation. Through width-based and depth-based extension, our framework constructs hierarchically complex challenges. Empirical results demonstrate its effectiveness in structured task generation, improving prompt optimization and supervised fine-tuning while reducing reliance on human annotation. Additionally, we release a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.

我们提出 TaskCraft，这是一套用于生成可扩展、支持多工具且可验证的智能体任务的自动化工作流。 通过基于宽度和基于深度的扩展，我们的框架构造出层级复杂的挑战。 实证结果证明了它在结构化任务生成方面的有效性：在减少对人工标注依赖的同时，改进提示优化和监督微调。 此外，我们发布一个包含约 36,000 个不同难度任务的大规模合成数据集，以支持未来关于智能体调优与评测的研究。

7. Limitation

This work currently focuses on constructing atomic tasks for common tools, including browsing, PDF processing, and image analysis. Future iterations will enable users to generate atomic tasks tailored to their agents' specific tool requirements.

本工作目前专注于为常用工具构造原子任务，包括网页浏览、PDF 处理和图像分析。 未来版本将支持用户根据其智能体的特定工具需求生成定制的原子任务。