AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents
AgentICLR 2026CCF-A加州大学伯克利分校AgentSynth：面向通用计算机使用智能体的可扩展任务生成
Abstract
We introduce AgentSynth, a scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer-use agents. Leveraging information asymmetry, AgentSynth constructs subtasks that are simple during generation but significantly more challenging when composed into long-horizon tasks, enabling the creation of over 6,000 diverse and realistic tasks. A key strength of AgentSynth is its ability to precisely modulate task complexity by varying the number of subtasks. Empirical evaluations show that state-of-the-art LLM agents suffer a steep performance drop, from 18% success at difficulty level 1 to just 4% at level 6, highlighting the benchmark's difficulty and discriminative power. Moreover, our pipeline achieves a low average cost of $0.60 per trajectory, orders of magnitude cheaper than human annotations.
我们提出 AgentSynth，这是一套可扩展且经济高效的流水线，用于自动合成面向通用计算机使用智能体的高质量任务与轨迹数据集。 AgentSynth 利用信息不对称，构造在生成时简单、但组合成长程任务后显著更具挑战性的子任务，从而能够创建 6,000 多个多样且真实的任务。 AgentSynth 的一项关键优势是，它能够通过改变子任务数量来精确调节任务复杂度。 实证评估表明，最先进的 LLM 智能体性能急剧下降：在难度 1 上的成功率为 18%，到难度 6 时仅为 4%，这突显了该基准的难度与区分能力。 此外，我们的流水线平均每条轨迹成本仅为 0.60 美元，比人工标注低几个数量级。
1. Introduction
Large language models (LLMs) have recently shown promise as autonomous agents capable of solving complex, multi-step tasks across a wide range of domains. These LLM agents interact with an environment through structured actions such as mouse clicks, keystrokes, or code executions, and are prompted to complete specific tasks using tools provided by the interface. This paradigm has been explored for web navigation tasks, software development, formal mathematics, and many others. As research in LLM agents progresses, the availability of high-quality datasets tailored to these domains becomes increasingly critical.
大语言模型（LLM）近来展现出作为自主智能体的潜力，能够解决广泛领域中的复杂多步任务。 这些 LLM 智能体通过鼠标点击、按键或代码执行等结构化动作与环境交互，并按照提示使用界面提供的工具完成特定任务。 这一范式已被用于网页导航任务、软件开发、形式化数学以及许多其他领域。 随着 LLM 智能体研究不断推进，面向这些领域的高质量数据集变得愈发关键。
General computer-use tasks that involve interacting with desktop environments and software applications pose especially difficult challenges for data collection. Existing datasets in this space, such as
需要与桌面环境和软件应用交互的通用计算机使用任务，给数据收集带来了尤为棘手的挑战。 该领域的现有数据集，如
To overcome these limitations, recent work has turned to synthetic data generation using LLMs. However, existing pipelines face two core challenges: (1) current LLM agents struggle to generate reliable trajectories for complex tasks, and (2) simplistic or repetitive generation strategies limit task diversity. These challenges are especially acute in visually grounded or long-horizon tasks, where agents must maintain contextual awareness, reason over multiple steps, and adapt when plans fail. Moreover, limited task diversity increases the risk of overfitting or model collapse during downstream training.
为了克服这些局限，近期工作开始转向使用 LLM 生成合成数据。 然而，现有流水线面临两项核心挑战：（1）当前 LLM 智能体难以为复杂任务生成可靠轨迹；（2）过于简单或重复的生成策略限制了任务多样性。 这些挑战在视觉落地任务或长程任务中尤其突出，因为智能体必须保持上下文感知、进行多步推理，并在计划失败时作出调整。 此外，有限的任务多样性会增加下游训练中过拟合或模型坍缩的风险。
We introduce AgentSynth, a scalable and flexible pipeline for synthesizing diverse, high-quality datasets for training and evaluating computer-use agents. The core insight behind AgentSynth is to exploit information asymmetry between the data generation and evaluation phases, the idea that solving a task step-by-step in the forward direction is far easier than reasoning out the entire solution all at once. Therefore, we construct the task through a sequence of simple, solvable subtasks. Each subtask builds incrementally on the prior state, with the corresponding trajectories collected during execution. A summarization agent then merges the subtasks into a composite long-horizon task, producing realistic scenarios that are easy to generate but hard to solve.
我们提出 AgentSynth，这是一套可扩展且灵活的流水线，用于合成多样的高质量数据集，以训练和评估计算机使用智能体。 AgentSynth 的核心洞见是利用数据生成阶段与评估阶段之间的信息不对称，即沿正向逐步解决任务远比一次性推断出完整解决方案容易。 因此，我们通过一系列简单且可解的子任务来构造任务。 每个子任务都在先前状态上逐步构建，同时在执行过程中收集相应轨迹。 随后，一个总结智能体将这些子任务合并为复合的长程任务，从而产生易于生成但难以求解的真实场景。
This design offers several key advantages. By constructing complex tasks from simple, solvable components, AgentSynth enables reliable trajectory collection while maintaining benchmark difficulty. Varying the chaining of subtasks induces combinatorial task diversity. The pipeline is fully automated and achieves a low cost of just $0.60 per trajectory. While we generate over 6,000 tasks in this work, the approach readily scales to tens of thousands of realistic tasks across diverse environments, unlocking significant potential for agent training and evaluation.
这一设计具有几项关键优势。 AgentSynth 从简单且可解的组件构造复杂任务，在保持基准难度的同时实现可靠的轨迹收集。 改变子任务的串联方式能够产生组合式任务多样性。 该流水线完全自动化，每条轨迹的成本仅为 0.60 美元。 虽然本工作生成了 6,000 多个任务，但该方法可以轻松扩展到不同环境中的数万个真实任务，为智能体训练与评估释放巨大潜力。
Our contributions are as follows:
- We introduce AgentSynth, a fully automated pipeline that synthesizes challenging and diverse computer-use tasks by iteratively chaining LLM-generated subtasks
- We demonstrate how information asymmetry between generation and execution improves trajectory reliability and task complexity, enabling fine-grained task difficulty control.
- We build a benchmark using AgentSynth and show that state-of-the-art agents struggle significantly, revealing a large room for future improvement.
我们的贡献如下：
- 我们提出 AgentSynth，这是一套完全自动化的流水线，通过迭代串联 LLM 生成的子任务来合成具有挑战性且多样的计算机使用任务。
- 我们展示了生成与执行之间的信息不对称如何提高轨迹可靠性与任务复杂度，并实现对任务难度的细粒度控制。
- 我们使用 AgentSynth 构建了一个基准，并表明最先进的智能体仍面临显著困难，揭示出巨大的未来改进空间。
2. Related Work
Substantial research has focused on synthesizing data to improve the training and evaluation of LLMs. However, most existing datasets and benchmarks for computer-use agents still rely heavily on manual design and annotation, limiting their scalability and diversity.
大量研究致力于合成数据，以改进 LLM 的训练与评估。 然而，现有的大多数计算机使用智能体数据集与基准仍严重依赖人工设计和标注，限制了其可扩展性与多样性。
Synthetic Data Generation. Synthetic data generation has emerged as a promising approach to enhance model performance and foster new capabilities. Many recent studies have leveraged LLMs to automate and diversify data generation. For instance, Yuan et al. curated diverse, high-quality datasets extracted from extensive pretraining corpora. Xu et al. (Evol-Instruct), Su et al. (Learn-by-interact), and Sun et al. (OS-Genesis) both use sequential pipelines to generate synthetic datasets. However, OS-Genesis and Learn-by-interact retroactively define a task over a trajectory instead of stringing together subtasks, while Evol-Instruct only generates the trajectory with the final instruction. Shin et al. generated synthetic datasets with controlled distributions over programs and specifications. Li et al. employed an optimization loop where a data generator continuously produces challenging problems targeted at specific evaluation models. Many other applications of synthetic data for LLMs are listed in Liu et al. These works highlight the power of synthetic pipelines but focus primarily on static text benchmarks rather than interactive agents.
合成数据生成。 合成数据生成已成为增强模型性能并培育新能力的一种有前景的方法。 近期许多研究利用 LLM 实现数据生成的自动化与多样化。 例如，Yuan 等人从大规模预训练语料库中整理出多样的高质量数据集。 Xu 等人（Evol-Instruct）、Su 等人（Learn-by-interact）和 Sun 等人（OS-Genesis）均使用顺序流水线生成合成数据集。 然而，OS-Genesis 和 Learn-by-interact 是在轨迹生成后回溯性地定义任务，而非串联子任务；Evol-Instruct 则只根据最终指令生成轨迹。 Shin 等人生成了程序与规约分布可控的合成数据集。 Li 等人采用一种优化循环，让数据生成器持续生成针对特定评估模型的挑战性问题。 Liu 等人列举了合成数据在 LLM 中的许多其他应用。 这些工作突显了合成流水线的能力，但主要聚焦静态文本基准，而非交互式智能体。
Agent Datasets and Benchmarks. Current datasets and benchmarks for agents predominantly depend on human annotators for task creation, demonstration provision, and the definition of evaluation metrics, which are costly to scale and often limited in diversity. More recent work explores using LLMs to generate agent tasks and trajectories. For example, Pahuja et al., Trabucco et al., Murty et al., and Gandhi et al. employed LLMs as web agents to synthesize web-based interactions. Boisvert et al. composed atomic tasks from Drouin et al. to form difficult tasks. Xu et al. and Ou et al. turned online tutorials into tasks and demonstrations. Nonetheless, these generated tasks and trajectories are limited primarily to web-based activities and typically involve simple interactions without complex multi-step reasoning or extensive tool utilization.
智能体数据集与基准。 当前的智能体数据集与基准主要依赖人工标注者来创建任务、提供演示并定义评估指标，这种方式扩展成本高昂，而且多样性通常有限。 较新的工作开始探索使用 LLM 生成智能体任务与轨迹。 例如，Pahuja 等人、Trabucco 等人、Murty 等人和 Gandhi 等人使用 LLM 作为网页智能体来合成基于网页的交互。 Boisvert 等人组合 Drouin 等人提出的原子任务，以形成困难任务。 Xu 等人和 Ou 等人将在线教程转化为任务与演示。 尽管如此，这些生成的任务与轨迹主要局限于网页活动，通常只涉及简单交互，缺少复杂的多步推理或广泛的工具使用。
Agent Environments. Early agent environments such as MiniWob++ focused on simplified web tasks and low-level actions. Later advancements such as Mind2Web, WebArena, and Online-Mind2Web introduced more realistic websites but remained constrained in breadth and complexity. More comprehensive environments have been developed by Yao et al., Drouin et al., and Xu et al. which expanded the action space and interface diversity, yet they still deviate from the actual computer environments. Recent developments like OSWorld and WindowsArena address this gap by transforming real operating systems into interactive gym environments for agent training and trajectory generation. Our work leverages the capabilities of OSWorld, providing comprehensive access to authentic computer tools to enhance synthetic data generation for generalist computer-use agents.
智能体环境。 MiniWob++ 等早期智能体环境专注于简化的网页任务和底层动作。 Mind2Web、WebArena 和 Online-Mind2Web 等后续工作引入了更真实的网站，但在广度与复杂度方面仍然受限。 Yao 等人、Drouin 等人和 Xu 等人开发了更全面的环境，扩展了动作空间与界面多样性，但这些环境仍与真实计算机环境存在差异。 OSWorld 和 WindowsArena 等近期工作通过将真实操作系统转化为用于智能体训练和轨迹生成的交互式 Gym 环境，弥合了这一差距。 我们的工作利用 OSWorld 的能力，全面访问真实计算机工具，从而增强面向通用计算机使用智能体的合成数据生成。
3. Scalable Agent Tasks and Trajectories Generation
We design a synthetic data generation pipeline powered by six distinct LLM-based agents: a task proposer, a task executor, a task verifier, a task reviser, a follow-up task proposer, and a task summarizer. Central to our methodology is the exploitation of information asymmetry; the idea that solving a task step-by-step in the forward direction is far easier than inferring the entire solution from scratch. Specifically, we generate sequences of simple, tractable subtasks, collecting trajectories along the way, and later summarizing the sequence into a single, coherent long-horizon task. This approach allows us to synthesize tasks that are easy to generate but substantially more difficult for agents to complete at test time. Full prompt templates are included in the appendix.
我们设计了一套由六个不同的 LLM 智能体驱动的合成数据生成流水线：任务提议智能体、任务执行智能体、任务验证智能体、任务修订智能体、后续任务提议智能体和任务总结智能体。 我们方法的核心是利用信息不对称，即沿正向逐步解决任务，远比从头推断完整解决方案容易。 具体而言，我们生成一系列简单且易处理的子任务，在此过程中收集轨迹，之后再将该序列总结为一个连贯的长程任务。 这种方法使我们能够合成易于生成、但智能体在测试时更难完成的任务。 完整的提示模板见附录。
Our pipeline operates in the OSWorld environment, a Gym-compatible simulated desktop interface that mirrors real-world computer usage. Within this environment, agents can interact freely with a broad range of software applications and system tools hosted on a virtual machine. At each step, the agent receives a full-screen screenshot (1920 pyautogui to closely emulate human behavior. The full action space is detailed in the appendix, where the percentage listed for each action type reflects its frequency of occurrence across all trajectories in our dataset. To highlight the generality of our pipeline, we also apply it to a web agent environment (InSTA), as discussed in the appendix. The overall data generation pipeline is detailed below and is presented in Figure 1.
我们的流水线运行于 OSWorld 环境中，这是一个兼容 Gym 的模拟桌面界面，用于复现现实世界的计算机使用方式。 在该环境中，智能体可以自由地与虚拟机上托管的各种软件应用和系统工具交互。 每一步中，智能体都会接收一张全屏截图（1920 pyautogui 执行，以逼真模拟人类行为。 完整动作空间见附录，其中每种动作类型的百分比表示它在我们数据集全部轨迹中的出现频率。 为了突出流水线的通用性，我们还将其应用于网页智能体环境 InSTA，相关内容见附录。 整体数据生成流水线详述如下，并展示于图1中。
(1) Task Proposer. We initiate the data generation process by instructing a task proposer agent to generate an initial, straightforward task. To enrich task diversity, the proposer is guided by a randomly assigned persona sampled from the persona hub, prompting it to suggest tasks relevant to a specific user profile. The proposer takes as input the persona and the initial Ubuntu desktop screenshot, and is prompted to create clear, specific tasks that can be completed in a few atomic actions. To ensure safety and privacy, we prohibit any tasks involving login credentials or actions such as email sending. Prompt details are provided in the appendix.
（1）任务提议智能体。 我们首先要求任务提议智能体生成一个初始的简单任务，从而启动数据生成过程。 为了丰富任务多样性，提议智能体由从 persona hub 中随机采样的用户画像引导，从而提出与特定用户特征相关的任务。 提议智能体以用户画像和初始 Ubuntu 桌面截图为输入，并按照提示创建清晰、具体且能用少量原子动作完成的任务。 为了确保安全与隐私，我们禁止任何涉及登录凭证或发送电子邮件等操作的任务。 提示详情见附录。
We currently rely on GPT-4.1-based agents for task generation due to their robustness and broad generalization. Different LLM models might generate tasks with systematically different complexity, realism, or meaningfulness, and it remains an open and interesting research question how model choice affects generated task properties. Exploring task-generation variance across different LLM architectures could be beneficial to further enrich task diversity and calibrate difficulty precisely.
由于 GPT-4.1 智能体具有稳健性和广泛的泛化能力，我们目前依赖它们生成任务。 不同 LLM 可能会生成在复杂度、真实性或意义上存在系统性差异的任务，而模型选择如何影响生成任务的属性仍是一个开放且有趣的研究问题。 探索不同 LLM 架构之间的任务生成差异，可能有助于进一步丰富任务多样性并精确校准难度。
(2) Task Executor. To execute the proposed tasks, we construct a ReAct-style agent that integrates OpenAI's GPT-4.1 and computer-use-preview models. Empirically, GPT-4.1 is good at planning and interpreting visual context, while the computer-use model is more accurate in grounding actions to pixel-level coordinates. We therefore assign GPT-4.1 the role of planner: it receives the task, current screenshot, and execution history, and outputs a natural language description of the next action. This description, along with the screenshot, is then passed to the computer-use model, which generates the precise executable action (e.g., mouse click coordinates, keystrokes). This two-stage setup balances high-level reasoning with fine-grained visual grounding. During execution, we log both the model’s reasoning trace and the resulting actions, enabling rich trajectory annotation. Each task execution is limited to a maximum of 10 steps. The prompts for the task executor are shown in the appendix.
（2）任务执行智能体。 为了执行提议的任务，我们构建了一个 ReAct 风格的智能体，集成 OpenAI 的 GPT-4.1 与 computer-use-preview 模型。 根据经验，GPT-4.1 擅长规划和解释视觉上下文，而计算机使用模型能更准确地将动作落地到像素级坐标。 因此，我们让 GPT-4.1 担任规划器：它接收任务、当前截图与执行历史，并输出下一步动作的自然语言描述。 随后，该描述连同截图一起传给计算机使用模型，由后者生成精确的可执行动作，例如鼠标点击坐标或按键。 这种两阶段设置在高层推理与细粒度视觉落地之间取得了平衡。 执行期间，我们同时记录模型的推理轨迹和最终动作，从而获得丰富的轨迹标注。 每次任务执行最多限制为 10 步。 任务执行智能体的提示见附录。
(3) Task Verifier. The task verification agent evaluates whether a given trajectory successfully completes the intended task. It reviews the full screenshot sequence and task description, and outputs both a binary success label and a completion percentage. To avoid overwhelming the verifier with excessive visual input, we adopt a WebJudge-style architecture inspired by Xue et al. The verifier first extracts key requirements from the task description, then analyzes each screenshot to select a subset of key screenshots most relevant to task completion. The final verdict is made based on the task description, identified key requirements, and the filtered key screenshots. To reduce token usage, all screenshots are downsampled to 960
（3）任务验证智能体。 任务验证智能体评估给定轨迹是否成功完成预期任务。 它审查完整截图序列和任务描述，并同时输出二元成功标签与完成百分比。 为了避免过多视觉输入使验证智能体不堪重负，我们采用受 Xue 等人启发的 WebJudge 风格架构。 验证智能体首先从任务描述中提取关键要求，随后分析每张截图，选出与任务完成最相关的一组关键截图。 最终判定依据任务描述、识别出的关键要求以及筛选后的关键截图作出。 为了减少 token 使用量，所有截图都被下采样至 960
(4) Task Reviser. When a trajectory is only partially successful, we invoke a task reviser agent to generate a revised task description that accurately reflects the actions actually completed by the agent. The reviser takes as input the full execution screenshots and identifies the goals that were successfully accomplished. It then outputs a revised task description that aligns with the observed behavior. Prompt details for the task reviser are shown in the appendix.
（4）任务修订智能体。 当一条轨迹只取得部分成功时，我们调用任务修订智能体生成修订后的任务描述，使其准确反映智能体实际完成的动作。 修订智能体以完整执行截图为输入，并识别已成功实现的目标。 随后，它输出与观测行为一致的修订任务描述。 任务修订智能体的提示详情见附录。
(5) Follow-up Task Proposer. Upon completing a task, the follow-up task proposer generates the next logical subtask to continue the sequence. This agent is given the full history of prior subtasks and the most recent desktop screenshot, and is instructed to generate a simple, specific follow-up action that builds on the previous state. Additionally, the proposer is informed of previously unsuccessful tasks, prompting it to propose simpler alternatives. Like the initial proposer, it avoids tasks that require login or unsafe actions. The resulting task is executed and verified as before, and if incomplete, a revised description is generated. This iterative generation process continues until a desired sequence length is reached. Prompt templates for the follow-up proposer are shown in the appendix.
（5）后续任务提议智能体。 完成一个任务后，后续任务提议智能体生成下一个合乎逻辑的子任务，以延续该序列。 该智能体接收先前子任务的完整历史与最新桌面截图，并按照指示生成一个简单、具体且建立在先前状态之上的后续动作。 此外，提议智能体还会获知先前失败的任务，从而促使它提出更简单的替代方案。 与初始提议智能体一样，它会避开需要登录或不安全操作的任务。 生成的任务会像之前一样被执行和验证；如果未完成，则生成修订后的描述。 这一迭代生成过程持续进行，直至达到所需的序列长度。 后续任务提议智能体的提示模板见附录。
(6) Task Summarizer. Finally, the task summarizer converts a sequence of completed subtasks into a single high-level task description. This summary abstracts away step-level details while preserving the overarching objective and required actions. By varying the number of subtasks summarized, we systematically control task difficulty: more subtasks yield longer, more complex tasks that require greater reasoning and planning. This mechanism enables us to generate tasks at multiple difficulty levels in a principled way. While each subtask may be trivial in isolation, the final composed task presents a challenging, multi-step problem for LLM agents. The summarization process is illustrated in Figure 1, and prompt details are provided in the appendix.
（6）任务总结智能体。 最后，任务总结智能体将一系列已完成的子任务转化为单个高层任务描述。 该总结抽象掉步骤级细节，同时保留总体目标和所需动作。 通过改变被总结的子任务数量，我们系统地控制任务难度：子任务越多，得到的任务越长、越复杂，需要的推理与规划也越多。 这一机制使我们能够以系统化方式生成多个难度级别的任务。 虽然每个子任务单独来看可能微不足道，但最终组合出的任务会给 LLM 智能体带来具有挑战性的多步问题。 总结过程展示于图1中，提示详情见附录。
4. Dataset Analysis
4.1 Quality
To assess the quality of the generated tasks and trajectories, we conducted a manual evaluation on a random sample of 100 instances across difficulty levels (approximately 16 tasks per difficulty level) to ensure representativeness across complexity. Our evaluation focused on the feasibility and realism of the overall task, the coherence and logical flow of subtasks, their relevance to the assigned persona, and the accuracy of the verifier’s assessment of the agent's trajectory. Specifically, human annotators are instructed to assess:
为了评估生成任务与轨迹的质量，我们从各难度级别随机抽取 100 个实例进行人工评估，每个难度级别约 16 个任务，以确保样本能够代表不同复杂度。 我们的评估重点包括整体任务的可行性与现实性、子任务的连贯性与逻辑流程、子任务与指定用户画像的相关性，以及验证智能体对智能体轨迹判断的准确性。 具体而言，人工标注者需要评估：
- Feasibility and realism: Could a real human user plausibly complete this task using standard software tools?
- Subtask coherence: Does each subtask logically follow from the previous subtasks, maintaining clear and meaningful workflow progression?
- Persona relevance: Is the task aligned meaningfully with the persona provided to guide task creation?
- Verifier accuracy: Does the automated verifier's binary assessment (task success or failure) align correctly with human judgment?
- 可行性与现实性： 真实人类用户能否使用标准软件工具合理地完成该任务？
- 子任务连贯性： 每个子任务是否在逻辑上承接先前的子任务，并保持清晰且有意义的工作流推进？
- 用户画像相关性： 任务是否与用于指导任务创建的用户画像具有实质性的关联？
- 验证准确性： 自动验证智能体的二元判断（任务成功或失败）是否与人工判断正确一致？
| Quality Metric | Yes |
|---|---|
| Feasibility and realism | 91% |
| Subtask coherence | 90% |
| Persona Relevance | 94% |
| Verifier Accuracy | 88% |
As shown in Table 1, all quality metrics exceed 85%, highlighting the consistency, realism, and reliability of the data produced by the AgentSynth pipeline. On verifier accuracy, the evaluators who independently evaluated the random sample also had an inter-rater agreement of 0.74 (Cohen's kappa). We note that prior findings indicate potential limitations of LLM-based verification, and our high manual-validation rate suggests our engineered verification pipeline, including selective screenshot sampling, visual context filtering, and task requirement extraction, improves verifier reliability compared to simpler methods.
如表1所示，所有质量指标均超过 85%，凸显了 AgentSynth 流水线所生成数据的一致性、现实性与可靠性。 在验证准确性方面，对随机样本进行独立评估的标注者之间一致性达到 0.74（Cohen's kappa）。 我们注意到，先前研究指出基于 LLM 的验证可能存在局限；而较高的人工验证通过率表明，我们设计的验证流水线包含选择性截图采样、视觉上下文筛选和任务要求提取，与更简单的方法相比提高了验证可靠性。
4.2 Verifier Calibration
We further calibrate our LLM-based verifier against human judgments on a stratified sample of trajectories. For each trajectory, humans label success/failure and assign a graded completion score in [0,1].
我们进一步在分层抽取的轨迹样本上，以人工判断校准基于 LLM 的验证智能体。 对于每条轨迹，人工标注者标记其成功或失败，并给出一个位于 [0,1] 区间内的分级完成度分数。
Figure 2 (left) reports binary agreement as a function of difficulty level. Accuracy remains high across all levels with only a mild decline as tasks become harder, indicating that the verifier's pass/fail decisions are consistent with human judgments.
图2左侧给出了二元判断一致率随难度级别的变化。 所有难度级别上的准确率都保持较高水平，仅随任务难度上升而略有下降，这表明验证智能体对通过或失败的判断与人工判断一致。
Figure 2 (right) examines the verifier's completion score. We bin trajectories by verifier score and plot the average human-judged completion score in each bin. The curve is monotone: higher verifier scores correspond to higher human scores. Together, these results show that the LLM verifier is both reliable across difficulty levels and provides a meaningful partial-credit signal.
图2右侧考察了验证智能体给出的完成度分数。 我们依据验证分数将轨迹分箱，并绘制每个分箱中人工判断的平均完成度分数。 该曲线呈单调变化：验证分数越高，对应的人工分数也越高。 综合来看，这些结果表明，LLM 验证智能体在不同难度级别上均具有可靠性，并且能够提供有意义的部分得分信号。
To further probe robustness, we perform an adversarial stress test of the LLM verifier. Starting from human-verified successful trajectories, we construct two types of perturbations: near-miss variants that subtly violate the goal (e.g., saving a file with an almost-correct name or in a wrong but visually similar folder), which should be labeled as failures, and benign variants that preserve the goal but change the UI in irrelevant ways (e.g., resized windows, extra tabs), which should still be labeled as successes. Table 2 reports the fraction of perturbed states that the verifier marks as “success.” The verifier incorrectly accepts only 12% of near-miss variants while correctly accepting 94% of benign variants, indicating a low false-positive rate on subtle failures and robustness to superficial UI changes.
为了进一步探查稳健性，我们对 LLM 验证智能体进行对抗性压力测试。 我们从经人工确认成功的轨迹出发，构造两类扰动：一类是细微违反目标的近失误变体，例如将文件保存为几乎正确的名称，或保存到错误但视觉上相似的文件夹中，这类变体应被标记为失败；另一类是保留目标、仅以无关方式改变界面的良性变体，例如调整窗口大小或增加标签页，这类变体仍应被标记为成功。 表2给出了被验证智能体标记为“成功”的扰动状态比例。 验证智能体仅错误接受了 12% 的近失误变体，同时正确接受了 94% 的良性变体，说明它对细微失败的误报率较低，并且对表层界面变化具有稳健性。
| Category | Verifier “success” |
|---|---|
| Near-miss (should fail) | 12% |
| Benign (should succeed) | 94% |
4.3 Comparison to Other Datasets and Benchmarks
We designed the AgentSynth pipeline with a focus on generating diverse, realistic, and challenging data for training and evaluating computer-use agents. Table 4 compares our dataset to several existing agent benchmarks, highlighting key advantages in diversity, complexity, and scalability. Examples of tasks are shown in the appendix.
我们在设计 AgentSynth 流水线时，着重为计算机使用智能体的训练与评估生成多样、现实且富有挑战性的数据。 表4将我们的数据集与若干现有智能体基准进行比较，突出了其在多样性、复杂度与可扩展性方面的关键优势。 任务示例见附录。
Diverse Real-World Tasks. AgentSynth spans a broad range of software applications and domains, including office productivity, information retrieval, entertainment, coding, and research. This breadth ensures rich task diversity and supports generalization across practical, everyday scenarios. The pipeline leverages versatile environments that require agents to fluidly interact with multiple software tools within a single task. Figure 3 (right) shows the coverage across domains and tools, illustrating the dataset's alignment with real-world complexity.
多样的现实世界任务。 AgentSynth 涵盖广泛的软件应用与领域，包括办公生产力、信息检索、娱乐、编程和研究。 这种广度确保了丰富的任务多样性，并支持模型在实用的日常场景中实现泛化。 该流水线利用多功能环境，要求智能体在单个任务中流畅地与多种软件工具交互。 图3右侧展示了对不同领域与工具的覆盖情况，说明该数据集与现实世界的复杂性相契合。
Importantly, our pipeline encourages multi-tool usage through chained subtasks. As shown in the appendix, over 60% of trajectories involve two or more software applications, and more than 40% involve three or more, demonstrating the inherent compositionality of AgentSynth tasks.
重要的是，我们的流水线通过串联子任务来鼓励使用多种工具。 如附录所示，超过 60% 的轨迹涉及两个或更多软件应用，超过 40% 涉及三个或更多软件应用，这体现了 AgentSynth 任务固有的组合性。
Long-Horizon Trajectories. Real-world tasks often require extended sequences of actions involving planning, memory, and interface coordination. AgentSynth explicitly supports such long-horizon tasks by composing them from interdependent subtasks. As shown in Figure 3, tasks at difficulty level 6 typically require 40-60 steps, exceeding the trajectory lengths of existing benchmarks. These tasks challenge agents to maintain context, manage interleaved goals, and execute multi-step plans, closely reflecting the demands of real-world computer use.
长程轨迹。 现实世界任务通常需要涉及规划、记忆与界面协调的长动作序列。 AgentSynth 通过组合相互依赖的子任务，明确支持此类长程任务。 如图3所示，难度级别 6 的任务通常需要 40–60 步，超过了现有基准中的轨迹长度。 这些任务要求智能体维持上下文、管理交错目标并执行多步计划，与现实世界计算机使用的需求高度一致。
Scaling of Task Complexity. Although we define difficulty primarily via horizon, the tasks become harder along several complementary axes. Figure 3 shows that from level 1 to 6, the average horizon increases from 5 to 45 steps and the memory span from 2 to 18 steps, indicating longer-range dependencies. Moreover, the average number of applications rises from 1.2 to 3.3 and app switches from 0.5 to 4.3, reflecting increased context switching and cross-application coordination. Thus, higher difficulty levels systematically combine longer horizons with more complex memory and interrupt-handling demands. Additional dataset statistics and task properties are provided in the appendix.
任务复杂度扩展。 虽然我们主要依据任务跨度定义难度，但任务还会沿多个互补维度变得更难。 图3显示，从级别 1 到 6，平均任务跨度由 5 步增至 45 步，记忆跨度由 2 步增至 18 步，这表明任务包含更长程的依赖。 此外，平均应用数量由 1.2 增至 3.3，应用切换次数由 0.5 增至 4.3，反映出更多的上下文切换与跨应用协调。 因此，更高的难度级别系统性地结合了更长的任务跨度、更复杂的记忆需求与中断处理需求。 更多数据集统计与任务属性见附录。
| Framework | Typical Steps | Human Hours per Task | Cost per Task |
|---|---|---|---|
| τ-bench | 20–30 | 2 | $4–$50 |
| OSWorld | 10–15 | 4.4 | $8.8–$110 |
| TheAgentCompany | 30–40 | 17 | $34–$425 |
| AgentSynth | 40–60 | NA | $0.6 |
| Framework | Multi-domain | Domain Categories | Scalable | Long Horizon |
|---|---|---|---|---|
| Mind2Web | × | Web | × | × |
| Online-Mind2Web | × | Web | × | × |
| WebArena | × | Web | × | × |
| VisualWebArena | × | Web | × | × |
| INSTA | × | Web | ✓ | × |
| AgentTrek | × | Web | ✓ | × |
| Explorer | × | Web | ✓ | ✓ |
| SWE-bench | × | Coding | × | × |
| WorkArena | ✓ | Enterprise Software | × | ✓ |
| OSWorld | ✓ | OS, Web, Office, Coding | × | × |
| WindowsAgentArena | ✓ | OS, Web, Office, Coding | × | × |
| τ-bench | ✓ | Retail, Airline | × | ✓ |
| TheAgentCompany | ✓ | SWE, HR, Admin, PM, Research | × | ✓ |
| AgentSynth | ✓ | Web, OS, Office, Coding, Research | ✓ | ✓ |
4.4 Cost Analysis
Beyond diversity and high quality, our data generation pipeline is also highly scalable and cost-efficient. Our approach achieves a cost of $0.6 per trajectory with 5 follow-up subtasks. This is comparable with recent methods such as AgentTrek ($0.55 per trajectory), Explorer ($0.28 per trajectory), and InSTA ($0.27 per trajectory). Furthermore, our method is much cheaper than human annotations for complex tasks with long trajectories. Table 3 shows the cost of several datasets from human annotations, where we assume the labor rate is in the range of $2–$25 per hour. The detailed calculation of our cost and human labor hours is shown in the appendix.
除了具有多样性和高质量外，我们的数据生成流水线还具有很强的可扩展性与成本效率。 在包含 5 个后续子任务时，我们的方法将每条轨迹的成本控制在 $0.6。 这一成本与近期方法相当，例如 AgentTrek 为每条轨迹 $0.55、Explorer 为 $0.28、InSTA 为 $0.27。 此外，对于轨迹较长的复杂任务，我们的方法比人工标注便宜得多。 表3给出了若干采用人工标注的数据集的成本，其中我们假设人工费率为每小时 $2–$25。 我们的成本与人工工时的详细计算见附录。
5. Results and Discussion
5.1 Evaluation Setup
To assess the general-purpose computer-use capabilities of current language models, we evaluated several state-of-the-art multimodal agents with visual understanding. At each interaction step, the model receives a prompt containing the task description, the current desktop screenshot, and its own previous thoughts. The model is then asked to generate executable Python code using the pyautogui library to perform the next action. We sampled 50 tasks from each difficulty level for agent evaluation. Additionally, to benchmark human performance, we evaluated 20 tasks sampled from difficulty level 6, the most challenging tier in AgentSynth.
为了评估当前语言模型的通用计算机使用能力，我们评估了若干具备视觉理解能力的先进多模态智能体。 在每个交互步骤中，模型都会收到一条包含任务描述、当前桌面截图及其先前思考的提示。 随后，模型需要使用 pyautogui 库生成可执行的 Python 代码，以执行下一步动作。 我们从每个难度级别抽取 50 个任务用于智能体评估。 此外，为了衡量人类表现，我们从 AgentSynth 中最具挑战性的难度级别 6 抽取 20 个任务进行评估。
To isolate the role of the underlying language model and focus on the task difficulty itself, we use bare LLMs without fine-tuning or additional agent-specific scaffolding. Each model is prompted to generate pyautogui actions step-by-step based on the screenshot, task description, and action histories. This setup reflects a lower bound on performance and is intended to benchmark agents under minimal guidance rather than deploy optimized, production-grade agents. Prompts used for evaluation are detailed in the appendix. Task completion is assessed using the automatic verifier agent introduced in Section 3, which analyzes the full trajectory and determines whether the task was successfully completed.
为了隔离底层语言模型的作用并聚焦任务难度本身，我们使用未经微调且没有额外智能体专用脚手架的裸 LLM。 每个模型都会根据截图、任务描述和动作历史，按照提示逐步生成 pyautogui 动作。 这一设置反映了性能下界，旨在衡量智能体在最少引导下的表现，而非部署经过优化的生产级智能体。 用于评估的提示详见附录。 任务完成情况由第 3 节介绍的自动验证智能体评估；该智能体分析完整轨迹，并判断任务是否成功完成。
5.2 Results
The top panel of Figure 4 shows the success rates of four state-of-the-art language models on the AgentSynth benchmark across task difficulty levels 1 through 6. Despite having visual capabilities and strong general reasoning skills, all models exhibit poor performance on our benchmark, especially as task complexity increases. In contrast, humans achieve a 70% success rate even on the most difficult tasks, underscoring the performance gap. Key observations include:
图4上半部分展示了四种先进语言模型在 AgentSynth 基准难度级别 1 至 6 上的成功率。 尽管这些模型具备视觉能力和强大的通用推理能力，但它们在我们的基准上表现不佳，尤其是在任务复杂度提高时。 相比之下，即使在最困难的任务上，人类也能达到 70% 的成功率，凸显了这一性能差距。 主要观察结果包括：
Sharp Decline with Difficulty. All models show a consistent and steep drop in success rate as task difficulty increases. For example, o4-mini achieves 18% success on level 1 but drops to 4% by levels 5 and 6. GPT-4.1 fails to complete tasks beyond level 3. This highlights the significant challenge in realistic GUI environments and demonstrates the increasing challenge posed by longer-horizon, multi-step tasks in AgentSynth.
成功率随难度急剧下降。 随着任务难度提高，所有模型的成功率都呈现一致且陡峭的下降。 例如，o4-mini 在级别 1 上的成功率为 18%，到级别 5 和 6 时降至 4%。 GPT-4.1 无法完成级别 3 以上的任务。 这凸显了现实 GUI 环境中的重大挑战，也表明 AgentSynth 中跨度更长的多步任务会带来越来越大的难度。
Near-Zero Success on Hard Tasks. At levels 4 and 6, only o4-mini and Claude-3.7 achieve non-zero scores, and even then, the success rate is only around 4%. This indicates that current models are far from achieving generalizable competence in realistic multi-step computer tasks, showcasing the difficulty and discriminative power of our benchmark. The results highlight the need for models that can handle long-term dependencies, maintain state, and ground their decisions in visual observations over extended sequences.
困难任务上的成功率接近于零。 在级别 4 和 6 上，只有 o4-mini 和 Claude-3.7 取得非零分数，即便如此，成功率也仅约为 4%。 这表明当前模型距离在现实多步计算机任务上获得可泛化能力仍然很远，也体现了我们基准的难度与区分能力。 这些结果凸显了对这样一类模型的需求：它们能够处理长期依赖、维持状态，并在长序列中依据视觉观测来落地决策。
5.3 Common Agent Failure Modes
Despite the promising capabilities of LLM agents, their performance on the AgentSynth benchmark remains low, with most tasks ending in failure. We identify several recurring failure modes that highlight key limitations and suggest directions for future improvement:
尽管 LLM 智能体展现出很有前景的能力，但它们在 AgentSynth 基准上的表现依然较低，多数任务最终以失败告终。 我们识别出若干反复出现的失败模式；这些模式揭示了关键局限，并为未来改进指出了方向：
Inaccurate Mouse Clicks. A frequent failure involves imprecise mouse click coordinates. While the agent often identifies the correct UI element conceptually (e.g., the “Save” button or a browser tab), it fails to locate it precisely on screen. This results in misclicks, unintended interactions (e.g., clicking ads or wrong icons), and cascading errors, such as obscuring or losing focus on the target window. Moreover, agents often repeat the same incorrect click multiple times without adapting.
鼠标点击不准确。 一种常见失败源于不精确的鼠标点击坐标。 智能体虽然往往能在概念上识别正确的界面元素，例如“保存”按钮或浏览器标签页，却无法在屏幕上准确定位它。 这会导致误点、非预期交互，例如点击广告或错误图标，以及一连串后续错误，例如遮挡目标窗口或使其失去焦点。 此外，智能体经常多次重复相同的错误点击，而不会进行调整。
Poor Screenshot Understanding and State Tracking. Agents frequently fail to properly interpret the visual information in screenshots. They may misidentify popups, ads, or irrelevant overlays as part of the main task UI. Other papers benchmarking LLM agents have also found problems with perceptual grounding. This weak perceptual grounding results in repetitive or irrational actions: for example, repeatedly trying to save a file that has already been saved. Moreover, agents often lose track of what has already been done, lacking persistent memory or state awareness.
截图理解与状态跟踪能力较差。 智能体经常无法正确解释截图中的视觉信息。 它们可能将弹窗、广告或无关覆盖层误认为主任务界面的一部分。 其他评测 LLM 智能体的论文同样发现了感知落地方面的问题。 这种薄弱的感知落地能力会导致重复或不合理的动作，例如反复尝试保存一个已经保存过的文件。 此外，智能体经常忘记已经完成的操作，缺乏持久记忆或状态感知。
Lack of Recovery from Errors. Once an agent becomes stuck, it struggles to recover. Rather than exploring alternative actions or reasoning about potential mistakes, the agent tends to repeat the same failed behavior. This lack of introspection and self-correction severely limits task completion, especially for multi-step tasks requiring contingency handling. This is a common error found with complex, long-horizon across many domains in the literature, from computer-use to general remote tasks to math and reasoning questions among others.
缺乏错误恢复能力。 智能体一旦陷入困境，就很难恢复。 智能体不会探索替代动作或推理潜在错误，而是倾向于重复相同的失败行为。 这种反思与自我纠正能力的缺失严重限制了任务完成，尤其是在需要处理意外情况的多步任务中。 文献在多个领域的复杂长程任务中都发现了这种常见错误，涉及计算机使用、通用远程任务、数学与推理问题等。
5.4 Agent scaffolding evaluation
Our primary results so far used bare LLM agents that directly map from the current screen and instruction to the next GUI action. To better match contemporary agentic systems, we additionally evaluate the Agent S3 scaffold, which augments the backbone model with explicit planning, tool-aware reasoning, and self-verification over intermediate steps. Figure 5 compares success rates across difficulty levels for four backbones. For the weaker models (o4-mini, GPT-4.1), adding the S3 scaffold yields substantial gains: success roughly doubles on level 1 and remains consistently higher than the bare agents across levels. Stronger backbones (GPT-5, GPT-5.1) under S3 achieve the highest absolute performance, confirming that both model quality and scaffolding contribute to success. However, for all backbones the success rate still decreases as the difficulty level increases, and even the best configuration attains only modest success on level 6. This pattern indicates that while sophisticated scaffolds can recover a significant fraction of “easy” tasks, AgentSynth remains challenging for state-of-the-art computer-use agents on the hardest levels.
到目前为止，我们的主要结果使用的是裸 LLM 智能体，它们将当前屏幕与指令直接映射为下一个 GUI 动作。 为了更贴近当代智能体系统，我们还评估了 Agent S3 脚手架；它为骨干模型加入显式规划、工具感知推理以及对中间步骤的自我验证。 图5比较了四种骨干模型在不同难度级别上的成功率。 对于较弱的模型（o4-mini、GPT-4.1），加入 S3 脚手架带来了显著提升：级别 1 上的成功率大致翻倍，并且在所有难度级别上始终高于裸智能体。 采用 S3 的更强骨干模型（GPT-5、GPT-5.1）取得了最高的绝对性能，证实模型质量与脚手架都会促进任务成功。 然而，对于所有骨干模型，成功率仍然会随难度级别提高而下降，即使最佳配置在级别 6 上也只取得有限的成功。 这一模式表明，尽管复杂脚手架能够挽回相当一部分“简单”任务，但在最困难的级别上，AgentSynth 对先进计算机使用智能体仍然具有挑战性。
We also evaluate an intermediate scaffold, Agent S2.5, which includes planning and tool formatting but uses a simpler self-check mechanism. As shown in the appendix, Agent S2.5 yields improvements over the bare agents that are similar to Agent S3, and exhibits a similar decline in success with increasing difficulty. This suggests the qualitative trends are robust to different scaffold designs.
我们还评估了一个中间级脚手架 Agent S2.5，它包含规划与工具格式化，但采用更简单的自检机制。 如附录所示，Agent S2.5 相比裸智能体带来的提升与 Agent S3 相近，并且成功率同样会随难度提高而下降。 这表明上述定性趋势对不同脚手架设计具有稳健性。
5.5 Effect of information asymmetry
To assess the impact of our asymmetric generation pipeline, we compare AgentSynth to a direct-instruction baseline. In the direct setting, we prompt the same base model to generate a single long-horizon instruction in one shot, targeting approximately 5-10, 10-20, and 20-30 actions, which we denote as Easy, Medium, and Hard tasks. We then attempt to execute each instruction in OSWorld with the same executor and verifier used in our main pipeline, and retain only tasks for which a complete ground-truth trajectory can be generated. In contrast, AgentSynth first generates and verifies short subtasks and only afterwards summarizes them into long-horizon instructions.
为了评估非对称生成流水线的影响，我们将 AgentSynth 与直接指令基线进行比较。 在直接设置中，我们提示同一个基础模型一次性生成一条长程指令，目标动作数分别约为 5–10、10–20 和 20–30，并将这些任务记为简单、中等和困难。 随后，我们使用主流水线中的同一个执行智能体与验证智能体，尝试在 OSWorld 中执行每条指令，并且只保留能够生成完整真实轨迹的任务。 相比之下，AgentSynth 首先生成并验证短子任务，之后才将它们总结为长程指令。
Table 5 reports both the generation success rate (fraction of tasks for which a ground-truth trajectory can be obtained) and the evaluation success rate of Agent S3 with GPT-5.1 on the resulting tasks. For direct instruction, generation success drops sharply as we target harder tasks: from 64% at Easy to only 11% at Hard, indicating that the generator frequently fails to produce trajectories for genuinely difficult tasks. However, among the small subset of tasks that are solvable, evaluation success remains relatively high and flat, suggesting that these retained tasks are still comparatively easy for the evaluation agent.
表5同时给出了生成成功率（能够获得真实轨迹的任务比例）以及采用 GPT-5.1 的 Agent S3 在所得任务上的评估成功率。 对于直接指令，随着目标任务变难，生成成功率急剧下降：从简单任务上的 64% 降至困难任务上的 11%，说明生成模型经常无法为真正困难的任务生成轨迹。 然而，在少量可解任务中，评估成功率依然相对较高且变化平缓，这说明这些被保留的任务对于评估智能体而言仍然比较简单。
AgentSynth exhibits the opposite pattern. Generation success remains high and stable across difficulty levels, confirming that the asymmetric pipeline makes trajectory collection substantially easier. At the same time, evaluation success drops dramatically as we increase the number of summarized subtasks, from 62% at level 1 to 14% at level 6. This decoupling between generation and evaluation difficulty is precisely the effect we seek: information asymmetry allows the generator to reliably solve and log simple multi-step workflows, while the resulting composite tasks remain challenging for downstream agents.
AgentSynth 呈现出相反的模式。 生成成功率在各难度级别上始终保持较高且稳定，这证实非对称流水线显著降低了轨迹收集的难度。 与此同时，随着被总结的子任务数量增加，评估成功率从级别 1 的 62% 急剧降至级别 6 的 14%。 生成难度与评估难度的这种解耦正是我们追求的效果：信息不对称使生成模型能够可靠地解决并记录简单的多步工作流，而最终得到的组合任务对下游智能体仍然具有挑战性。
| Direct instruction | AgentSynth | |||||
|---|---|---|---|---|---|---|
| Easy | Medium | Hard | Level 1 | Level 3 | Level 6 | |
| Generation success rate (%) | 64 | 33 | 11 | 65 | 57 | 52 |
| Evaluation success rate (%) | 60 | 51 | 48 | 62 | 20 | 14 |
5.6 Effect of base generator
Our main experiments so far used GPT-4.1 as the base model for proposing and executing subtasks during data synthesis. To assess how sensitive AgentSynth is to this choice, we re-run the full pipeline with three different base models: GPT-5-mini, GPT-5, and GPT-5.1, keeping prompts and the pipeline fixed. We then evaluate the resulting datasets using Agent S3 framework with either GPT-4.1 or GPT-5.1 as the backbone. Table 6 reports success rates across difficulty levels for each combination.
到目前为止，我们的主要实验使用 GPT-4.1 作为数据合成期间提议与执行子任务的基础模型。 为了评估 AgentSynth 对这一选择的敏感性，我们在保持提示和流水线不变的情况下，分别使用三种不同的基础模型重新运行完整流水线：GPT-5-mini、GPT-5 和 GPT-5.1。 随后，我们使用 Agent S3 框架评估所得数据集，并分别采用 GPT-4.1 或 GPT-5.1 作为骨干模型。 表6给出了每种组合在各难度级别上的成功率。
We observe that the overall behavior is stable across generation base models. Within each dataset, the stronger evaluation model (GPT-5.1 + S3) consistently outperforms GPT-4.1 + S3, and in all cases success rates decrease as task difficulty increases. Changing the base generator from GPT-4.1 to GPT-5-mini yields very similar results, while using GPT-5 or GPT-5.1 for synthesis leads to slightly lower success rates, suggesting that stronger generators may produce harder or more diverse tasks. These results remain consistent with the previous conclusions qualitatively. AgentSynth remains challenging at high difficulty for various generator choices, and the relative ranking of evaluation models is preserved. This indicates that our findings are not specific to a particular base model and that AgentSynth is robust to reasonable variations in the synthesis backbone.
我们观察到，不同生成基础模型的整体行为都很稳定。 在每个数据集中，更强的评估模型（GPT-5.1 + S3）始终优于 GPT-4.1 + S3，并且所有情况下的成功率都会随任务难度提高而下降。 将基础生成模型从 GPT-4.1 更换为 GPT-5-mini 会得到非常相似的结果，而使用 GPT-5 或 GPT-5.1 进行合成会使成功率略低，这表明更强的生成模型可能会产生更困难或更多样的任务。 这些结果在定性上仍与先前结论一致。 无论选择何种生成模型，AgentSynth 在高难度下都仍然具有挑战性，并且评估模型的相对排名保持不变。 这表明我们的发现并非某个特定基础模型所独有，同时说明 AgentSynth 对合成骨干模型的合理变化具有稳健性。
| Gen. model | Eval. model | Success rate (%) at difficulty level | |||||
|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | ||
| GPT-4.1 | GPT-4.1 + S3 | 36 | 18 | 14 | 14 | 10 | 10 |
| GPT-5.1 + S3 | 62 | 32 | 20 | 22 | 16 | 14 | |
| GPT-5-mini | GPT-4.1 + S3 | 34 | 16 | 16 | 14 | 10 | 12 |
| GPT-5.1 + S3 | 60 | 34 | 18 | 20 | 18 | 16 | |
| GPT-5 | GPT-4.1 + S3 | 28 | 12 | 14 | 10 | 8 | 6 |
| GPT-5.1 + S3 | 58 | 28 | 18 | 16 | 14 | 14 | |
| GPT-5.1 | GPT-4.1 + S3 | 30 | 14 | 12 | 12 | 10 | 8 |
| GPT-5.1 + S3 | 56 | 30 | 16 | 18 | 14 | 12 | |
6. Conclusion
In this work, we introduce AgentSynth, a scalable pipeline for synthesizing diverse, high-quality datasets of computer-use tasks and trajectories. By leveraging information asymmetry and LLM-based agents, our method decomposes complex tasks into solvable subtasks, enabling fine-grained control over difficulty and long-horizon planning. The resulting benchmark reveals performance gaps in state-of-the-art models, with success rates dropping sharply as task complexity increases, highlighting both the challenging and discriminative power of our dataset. Notably, AgentSynth can continuously and flexibly generate harder tasks, ensuring long-term benchmarking utility.
在本工作中，我们提出 AgentSynth，这是一套可扩展的流水线，用于合成多样且高质量的计算机使用任务与轨迹数据集。 通过利用信息不对称与基于 LLM 的智能体，我们的方法将复杂任务分解为可解的子任务，从而能够对难度和长程规划进行细粒度控制。 所得基准揭示了先进模型的性能差距：成功率会随任务复杂度提高而急剧下降，凸显了我们数据集的挑战性与区分能力。 值得注意的是，AgentSynth 能够持续且灵活地生成更困难的任务，从而确保其长期基准测试价值。
Ethics Statement
We acknowledge that we adhere to the ICLR Code of Ethics. Our data synthesis pipeline is intended solely for academic research on GUI-based computer-use agents. To ensure ethical integrity and mitigate potential risks, we explicitly prompt our proposer agents to avoid any tasks involving login credentials or real personal information, or sending forms that change the backend of any website (emails, transactions, etc). Additionally, we introduce time delays between each action to prevent excessive requests and reduce potential load on external websites, thereby minimizing any unintended impact on online services. Finally, only secure applications and websites are visited so that users of our dataset do not run into malware or other security issues. We also recognize that our method could be used to improve LLMs' abilities in harmful tasks like cyberattacks.
我们确认遵守 ICLR 伦理准则。 我们的数据合成流水线仅用于面向 GUI 的计算机使用智能体学术研究。 为了确保伦理合规并降低潜在风险，我们明确提示任务提议智能体避开任何涉及登录凭据、真实个人信息或提交会改变网站后端状态的表单的任务，例如电子邮件和交易。 此外，我们在每个动作之间加入时间延迟，以防止请求过于频繁并减轻外部网站的潜在负载，从而尽可能减少对在线服务的意外影响。 最后，智能体只会访问安全的应用和网站，以免数据集用户遭遇恶意软件或其他安全问题。 我们也认识到，该方法可能被用于提升 LLM 执行网络攻击等有害任务的能力。
Reproducibility Statement
We took several steps to facilitate reproduction of our results. The end-to-end pipeline, agent roles, and OSWorld environment setup are described in Section 3, including implementation details such as screenshot resolution, action execution, and verification flow. Complete prompt templates for every component and for the evaluation agent are provided in the appendix, and the OS-level action space used by agents is specified in the appendix. Dataset construction, difficulty formation, statistics, and our evaluation protocol that support the main experiments are reported in Section 5. We also document the cost accounting required to replicate our scale estimates in the appendix. We include anonymized source code and a minimal example dataset in the supplementary materials, and we will release both the full codebase and dataset publicly after the double-blind review process.