---

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

MemoryAgentBenchmarkLong ContextAgentRunbook100+加州大学洛杉矶分校

Wu D, Ji Z, Kawatkar A, et al. LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues[EB/OL]. arXiv:2605.12493, 2026.

https://xiaowu0162.github.io/longmemeval-v2/

https://github.com/xiaowu0162/LongMemEval-V2

---

LongMemEval-V2：面向经验丰富同事的长期智能体记忆评估

Abstract

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments. LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness.

长期记忆对于专用网络环境中的智能体至关重要，因为成功取决于能否回忆界面功能、状态动态、工作流以及反复出现的失败模式。 然而，现有智能体记忆基准大多关注用户历史、短轨迹或下游任务成功率，尚未回答如何直接评估记忆系统是否有效内化了特定环境经验。 为弥补这一空白，我们提出 LongMemEval-V2（LME-V2），用于评估记忆系统能否帮助智能体获得在定制环境中成为知识丰富同事所需的经验。 LME-V2 包含 451 个人工整理的问题，覆盖网络智能体的五项核心记忆能力：静态状态回忆、动态状态追踪、工作流知识、环境陷阱和前提感知。

These questions are paired with history trajectories containing up to 500 trajectories and 115M tokens. We use a context gathering formulation: memory systems consume history trajectories and return compact evidence for downstream question answering. As initial baselines for this challenging setting, we propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show that AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and the off-the-shelf coding agent baseline (69.3%).

这些问题与最长包含 500 条轨迹和 1.15 亿 token 的历史轨迹配对。 我们采用上下文收集形式：记忆系统读取历史轨迹，并为下游问答返回紧凑证据。 针对这一具有挑战性的设置，我们提出两种初始记忆基线：AgentRunbook-R 是一种高效的 RAG 记忆，分别为原始状态观察、事件和策略笔记建立知识池；AgentRunbook-C 则把轨迹存为文件，并调用编码智能体在增强沙箱中收集证据。 实验表明，AgentRunbook-C 取得最佳表现，平均准确率为 72.5%，超过最强 RAG 基线（48.5%）和开箱即用的编码智能体基线（69.3%）。

Despite the strong performance gains, coding agent based methods have high latency costs. While AgentRunbook-C advances the accuracy-latency Pareto frontier, substantial room for improvement remains. Together, these results establish LME-V2 as a challenging testbed for developing long-term memory systems that turn accumulated agent trajectories into reusable environment experience.

尽管性能提升显著，基于编码智能体的方法仍具有很高的延迟成本。 虽然 AgentRunbook-C 推进了准确率-延迟帕累托前沿，但仍存在很大的改进空间。 综合来看，这些结果确立了 LME-V2 作为一项富有挑战性的试验平台，可用于开发将累积智能体轨迹转化为可复用环境经验的长期记忆系统。

1. Introduction

Long-term memory helps large language models (LLMs) operate beyond their context and parameters by storing and recalling information over long horizons. Memory is especially important for agent systems, where LLMs interact with specialized environments over many steps. Recent works show that memorizing task procedures, interface affordances, and hidden failure modes improve agent performance at inference time.

长期记忆通过跨越长时间范围存储和回忆信息，帮助大语言模型（LLM）突破上下文与参数的限制运行。 记忆对于智能体系统尤其重要，因为 LLM 会经过许多步骤与专用环境交互。 近期工作表明，记住任务流程、界面功能和隐藏的失败模式可以提高智能体的推理时性能。

However, benchmarks for memory in the agentic context remain limited. Existing memory works mainly evaluate retrieval and reasoning over long documents or user chat histories. Recent works consider evaluating memorization over agent trajectories, but often use simplified game environments, emphasize limited dependencies within one or a few trajectories, or evaluate indirectly through downstream task success. As a result, they provide limited insight into whether memory systems can accumulate holistic, environment-specific knowledge from sustained interaction with a complex environment. To highlight this perspective, this paper uses the following framing:

然而，智能体语境下的记忆基准仍然有限。 现有记忆工作主要评估在长文档或用户聊天历史上的检索与推理。 近期工作开始考虑评估智能体轨迹上的记忆，但往往使用简化的游戏环境，强调一条或少数几条轨迹中的有限依赖关系，或者通过下游任务成功率间接评估。 因此，它们难以揭示记忆系统能否通过与复杂环境的持续交互，积累整体性的特定环境知识。 为了突出这一视角，本文采用如下表述：

A high-quality memory makes an agent an experienced colleague in a specialized environment.高质量记忆使智能体成为专用环境中经验丰富的同事。

Driven by this view, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help web agents acquire the experience needed to become knowledgeable colleagues. LME-V2 leverages customized websites including Magento shopping, shopping admin, Postmill forum, and ServiceNow from WebArena and WorkArena. From task-solving web agent trajectories, we manually curate 451 questions covering five core memory abilities: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. We provide examples in Figure 1 and ability definitions in Section 3.1. These questions are specific to the customized environments and thus remain generally unanswerable by recent frontier LLMs (Section 3.4).

在这一观点的驱动下，我们提出 LongMemEval-V2（LME-V2），用于评估记忆系统能否帮助网络智能体获得成为知识丰富同事所需的经验。 LME-V2 使用 WebArena 和 WorkArena 中的定制网站，包括 Magento 购物、购物管理、Postmill 论坛和 ServiceNow。 我们从解决任务的网络智能体轨迹中人工整理了 451 个问题，覆盖五项核心记忆能力：静态状态回忆、动态状态追踪、工作流知识、环境陷阱和前提感知。 示例见图1，能力定义见第 3.1 节。 这些问题针对定制环境，因此近期前沿 LLM 通常无法回答（第 3.4 节）。

LME-V2 further pairs the questions with a sequence of web agent trajectories ("haystacks", following Kamradt), where only a small fraction bears the answers to each question ("needles"). LME-V2-Small provides a 100-trajectory haystack shared by all questions, and LME-V2-Medium has 500-trajectory question-specific haystacks. Compared to prior benchmarks, LME-V2 poses new challenges with its deep context (25M/115M tokens in the small/medium tiers) and comprehensive memory ability coverage (Table 1).

LME-V2 还为问题配备一系列网络智能体轨迹，即沿用 Kamradt 表述的“干草堆”；其中只有一小部分轨迹包含每个问题的答案，即“针”。 LME-V2-Small 提供由所有问题共享的 100 条轨迹干草堆，LME-V2-Medium 则为每个问题提供包含约 500 条轨迹的专用干草堆。 与先前基准相比，LME-V2 通过深层上下文（小型/中型层级分别为 2500 万/1.15 亿 token）和全面的记忆能力覆盖带来了新挑战（见表1）。

图1：LongMemEval-V2 问题示例。图中展示 WorkArena 领域的示例，分别考察静态状态回忆（第 1 行）、动态状态追踪（第 2 行）、工作流知识（第 3 行）、环境陷阱（第 4 行）以及前提感知（右列）。

LME-V2 evaluates memory with a context gathering formulation (Section 3.3). A memory system implements two APIs: Insert, which consumes a trajectory, and Query, which returns a multimodal memory context for a question. For each question, we stream the associated trajectories sequentially into memory, invoke Query, truncate the returned context to a fixed token budget, and ask a fixed reader LLM to answer. This provides a direct evaluation of memory quality with a practical interface that a downstream agent would use. We report both answer accuracy and query latency.

LME-V2 采用上下文收集形式评估记忆（第 3.3 节）。 记忆系统实现两个 API：Insert 读取一条轨迹，Query 为问题返回多模态记忆上下文。 对于每个问题，我们把关联轨迹依次流式写入记忆，调用 Query，把返回的上下文截断到固定 token 预算，然后让固定的阅读器 LLM 作答。 这种设置通过下游智能体会实际使用的接口，直接评估记忆质量。 我们同时报告答案准确率和查询延迟。

表1：与现有记忆和长上下文基准的比较。# Sess.、# Tok. 和 # Q 分别表示历史会话数、历史 token 数和问题总数；MM 表示上下文或问题是否混合文本与图像。对于具有多个预设长度层级的基准，表中报告最短与最长层级之间的范围；其余基准报告全部样本的平均上下文规模。

Benchmark	Domain	Context Profile			Question Profile		Memory Ability
		# Sess.	# Tok.	MM	# Q	MM	Static	Dynamic	Workflows	Gotchas	Premise
General Long Context
LongBench V2	Mixed	N/A	260k	×	503	×	✓	×	×	×	×
MemoryAgentBench	Mixed	N/A	285k	×	2,071	×	✓	×	×	×	×
CL-Bench	Mixed	N/A	10k	×	1,899	×	✓	×	✓	×	×
Conversational Long Context
LoCoMo	User-user chat	28	∼ 16k	✓	7,512	×	✓	✓	×	×	✓
LongMemEval-V1	User-assistant chat	48–475	115k–1.5M	×	500	×	✓	✓	×	×	✓
PersonaMem	User-assistant chat	5–60	26k–951k	×	5,990	×	✓	✓	×	×	×
PersonaMem-v2	User-assistant chat	10–20	33k–124k	✓	5,000	×	✓	✓	×	×	×
BEAM	User-assistant chat	4.5–100	124k–10M	×	2,000	×	✓	✓	×	×	✓
Agentic Long Context
MemoryArena	Agent (mixed)	7	40k+	×	766	×	✓	×	✓	✓	×
AgentLongBench	Game agent	1	31k–4M	×	6,400	×	✓	✓	×	×	×
EMemBench	Game agent	1	2k–∞	✓	1,280+	×	✓	×	✓	✓	✓
FileGramBench	File-system agent	12	11k	✓	4,333	×	✓	×	✓	×	×
AMA-Bench	Agent (mixed)	1	57k	×	2,496	×	✓	✓	✓	✓	×
LongMemEval-V2	Web agent	100–498	25M–115M	✓	451	✓	✓	✓	✓	✓	✓

To succeed in LME-V2, a memory system needs to intelligently store and filter information from the noisy agent trajectories, retaining both low-level observations as well as higher-level environment dynamics and procedural knowledge. As a result, naive application of popular agent memory methods could be ineffective as they are biased towards less noisy conversational contexts or high-level strategic knowledge. In this paper, we propose AgentRunbook, a simple yet effective baseline consisting of two variants, optimized separately for efficiency and accuracy. AgentRunbook-R is an efficient retrieval-augmented generation (RAG) pipeline inspired by agentic memory works such as Xu et al.

要在 LME-V2 上取得成功，记忆系统需要从嘈杂的智能体轨迹中智能地存储和筛选信息，同时保留低层观察以及更高层的环境动态和流程知识。 因此，直接套用流行的智能体记忆方法可能效果不佳，因为它们偏向噪声较少的对话上下文或高层策略知识。 在本文中，我们提出 AgentRunbook，这是一项简单而有效的基线，由两个分别针对效率和准确率优化的变体组成。 AgentRunbook-R 是一条高效的检索增强生成（RAG）流程，其灵感来自 Xu 等人的智能体记忆工作。

It prompts an LLM controller to update and to actively query three knowledge pools: raw observations, state transition events, and high-level strategy notes (Section 4.1). AgentRunbook-R is efficient and covers major memory abilities, but its simple design is not optimized for detailed evidence selection. Inspired by Cao et al., we propose AgentRunbook-C, a coding agent-based memory method that casts memory management as a file management problem. AgentRunbook-C stores raw trajectories directly as files. At query time, it augments an off-the-shelf coding agent harness with workflow documents, memory manifests, and helper scripts, then invokes the agent to assemble a compact evidence set (Section 4.2).

它提示 LLM 控制器更新并主动查询三个知识池：原始观察、状态转换事件和高层策略笔记（第 4.1 节）。 AgentRunbook-R 效率高并覆盖主要记忆能力，但其简单设计并未针对细粒度证据选择进行优化。 受 Cao 等人启发，我们提出 AgentRunbook-C，这是一种基于编码智能体的记忆方法，把记忆管理转化为文件管理问题。 AgentRunbook-C 直接把原始轨迹存储为文件。 查询时，它用工作流文档、记忆清单和辅助脚本增强开箱即用的编码智能体框架，然后调用智能体组装紧凑的证据集（第 4.2 节）。

We evaluate the memory designs on the small and medium tiers of LME-V2. To begin with, a simple RAG method that retrieves state slices can only achieve an overall acccuracy of 40.1%, and AgentRunbook-R further improves to 57.8%. Accuracy-wise, we find the off-the-shelf Codex agent has competitive performance, achieving a surprisingly high 69.3% accuracy. However, the agent achieves this at a cost of about 182 seconds per query, about 6.9 times slower than AgentRunbook-R.

我们在 LME-V2 的小型和中型层级上评估这些记忆设计。 首先，只检索状态切片的简单 RAG 方法总体准确率仅为 40.1%，AgentRunbook-R 则进一步提高到 57.8%。 就准确率而言，我们发现开箱即用的 Codex 智能体具有竞争力，达到令人意外的 69.3%。 然而，这一表现的代价是每次查询约 182 秒，比 AgentRunbook-R 慢约 6.9 倍。

With our specialization designs, AgentRunbook-C performs best overall with 72.5% accuracy while being 32% faster than Codex at query time. Our further analyses reveal that AgentRunbook-C significantly advances the accuracy-latency frontier, but the room for future improvement remains large (Section 5.2). Overall, LME-V2 formulates a new standard for agent memory evaluation and provides a concrete testbed for memory modules that make long-running agents more reliable, adaptive, and useful in real-world environments.

借助这些专用设计，AgentRunbook-C 以 72.5% 的准确率取得最佳总体表现，同时查询速度比 Codex 快 32%。 进一步分析表明，AgentRunbook-C 显著推进了准确率-延迟前沿，但未来仍有很大的改进空间（第 5.2 节）。 总体而言，LME-V2 为智能体记忆评估制定了新标准，并为开发使长期运行智能体在真实环境中更可靠、更具适应性且更有用的记忆模块提供了具体试验平台。

2. Related Work

Long-Context and Memory Evaluation. Long-term memory evaluation can be seen as part of the long-lasting effort to evaluate LLMs and retrieval systems on recalling information across extended context. Early line of benchmarks focuses on testing information retrieval, aggregation, and instruction following over long input documents. Subsequent work expanded the focus to personalized memory, covering explicit user facts and implicit preferences, with benchmarks such as LoCoMo, DialSim, PerLTQA, LongMemEval, PersonaMem, and BEAM. LMEB isolates the retrieval component and evaluates dense retrievers on memory workloads. In contrast, among a new series of efforts, LME-V2 targets experience memory with context constructed from web agent history trajectories. This shift introduces substantially more complex contexts, a new ability taxonomy, and memory designs centered on agent experience.

长上下文与记忆评估。 长期记忆评估可以视为长期努力的一部分，这些工作旨在评估 LLM 和检索系统从扩展上下文中回忆信息的能力。 早期一系列基准主要测试在长输入文档上的信息检索、聚合和指令遵循。 后续工作把关注点扩展到个性化记忆，覆盖显式用户事实和隐式偏好，代表性基准包括 LoCoMo、DialSim、PerLTQA、LongMemEval、PersonaMem 和 BEAM。 LMEB 隔离检索组件，并在记忆工作负载上评估稠密检索器。 相比之下，在一系列新工作中，LME-V2 面向经验记忆，其上下文由网络智能体历史轨迹构成。 这一转变引入了复杂得多的上下文、新的能力分类体系，以及以智能体经验为中心的记忆设计。

Memory Systems for Agents. As LLM agents tackle long-horizon tasks in complex environments, memory becomes important both for recalling earlier detailed trajectory context and for consolidating high-level knowledge across trajectories. Memory has also been linked to improving inference-time performance through extended exploration and sleep-time offline consolidation. Despite this progress, direct evaluation of memory quality in agent settings remains limited. MemoryArena measures memory indirectly through the success rate of interdependent task sequences. AgentLongBench and EMemBench use synthetic agent histories and test recall of details from those traces. FileGram studies reasoning over file system behavior traces.

智能体记忆系统。 随着 LLM 智能体在复杂环境中处理长程任务，记忆既对回忆较早的详细轨迹上下文十分重要，也有助于跨轨迹整合高层知识。 记忆还被用于通过扩展探索和休眠期离线整合提高推理时性能。 尽管取得了这些进展，智能体场景中对记忆质量的直接评估仍然有限。 MemoryArena 通过相互依赖任务序列的成功率间接衡量记忆。 AgentLongBench 和 EMemBench 使用合成智能体历史，并测试对这些轨迹细节的回忆。 FileGram 研究对文件系统行为轨迹的推理。

AMA-Bench is closest to our setting, as it curates questions from agent trajectories in diverse domains such as embodied, web, and gaming agents. However, AMA-Bench focuses on understanding one trajectory, while LME-V2 focuses on environment knowledge induced across many past trajectories. To our knowledge, LME-V2 is also the first benchmark in this setting to scale the history length to tens or even over 100 million tokens.

AMA-Bench 与我们的设置最接近，因为它从具身智能体、网络智能体和游戏智能体等不同领域的智能体轨迹中整理问题。 然而，AMA-Bench 关注理解单条轨迹，而 LME-V2 关注从大量过去轨迹中归纳出的环境知识。 据我们所知，LME-V2 还是这一设置中首个把历史长度扩展到数千万乃至超过 1 亿 token 的基准。

Agents as Memory Controllers. Recent work on agentic memory proposes memory systems in which memory write and read operations are controlled by an LLM rather than a fixed pipeline. MemGPT and StateLM enable models to manage context programmatically. A-MEM and Mem0 introduce scaffolding that allows an LLM to evolve memory content and structure over time. Memory-R1 and Mem-$\alpha$ learn memory update actions via reinforcement learning. MemSkill learns memory skills to guide memory update behavior at a finer granularity. In this work, we further expand the notion of agentic memory.

作为记忆控制器的智能体。 近期智能体记忆工作提出由 LLM 而不是固定流程控制记忆读写操作的记忆系统。 MemGPT 和 StateLM 使模型能够以编程方式管理上下文。 A-MEM 和 Mem0 引入脚手架，使 LLM 能够随时间演化记忆内容和结构。 Memory-R1 和 Mem-$\alpha$ 通过强化学习获得记忆更新动作。 MemSkill 学习记忆技能，以更细粒度地指导记忆更新行为。 在本工作中，我们进一步扩展智能体记忆的概念。

Inspired by Cao et al. and CocoaBench Team et al., we view a general coding agent with tool use and file system manipulation abilities as a strong controller for file-based memory. Based on this perspective, we design AgentRunbook-C, which augments an off-the-shelf coding agent harness with workflow documents, query-time rendered artifacts, and helper scripts, yielding a strong accuracy-latency trade-off on LME-V2.

受 Cao 等人和 CocoaBench 团队等人的启发，我们把具备工具使用和文件系统操作能力的通用编码智能体视为文件式记忆的强大控制器。 基于这一视角，我们设计 AgentRunbook-C，用工作流文档、查询时渲染的工件和辅助脚本增强开箱即用的编码智能体框架，在 LME-V2 上取得良好的准确率-延迟权衡。

3. LongMemEval-V2

3.1 Core Memory Ability Definition

What does an experienced colleague internalize after repeatedly working in an environment? We categorize the learned experience into five memory abilities:

一位经验丰富的同事在某个环境中反复工作后，会内化哪些内容？ 我们把学到的经验分为五项记忆能力：

• Static State Recall. An experienced colleague remembers important landmarks, page layouts, module affordances, and subtle differences across states.
• Dynamic State Tracking. An experienced colleague can act as a world model of the environment: given states and actions, they understand how the environment changes.
• Workflow Knowledge. An experienced colleague knows the steps needed to perform common tasks in the customized environment.
• Environment Gotchas. An experienced colleague is aware of common recurring issues in the current environment and can avoid environment-specific failures.
• Premise Awareness. An experienced colleague can recognize assumptions that are valid in another environment but wrong in the current one.

• 静态状态回忆。 经验丰富的同事会记住重要地标、页面布局、模块功能以及不同状态之间的细微差异。
• 动态状态追踪。 经验丰富的同事可以充当环境的世界模型：给定状态和动作，他们能够理解环境如何变化。
• 工作流知识。 经验丰富的同事知道在定制环境中执行常见任务所需的步骤。
• 环境陷阱。 经验丰富的同事了解当前环境中反复出现的常见问题，并能避免特定环境中的失败。
• 前提感知。 经验丰富的同事能够识别在另一个环境中有效、但在当前环境中错误的假设。

3.2 Annotation

To holistically evaluate these memory abilities, we curate LongMemEval-V2 from multimodal web agent trajectories. The annotation has four steps: trajectory collection, question annotation, answer trajectory labeling, and haystack creation. We present full details in the appendix.

为了整体评估这些记忆能力，我们从多模态网络智能体轨迹中整理 LongMemEval-V2。 标注分为四个步骤：轨迹收集、问题标注、答案轨迹标记和干草堆构建。 完整细节见附录。

Trajectory Collection. We collect trajectories from three web agent benchmarks: WebArena, WorkArena, and WorkArena++, leveraging their OneStopShop, CMS, Reddit, ServiceNow environments. The trajectories are collected using the AgentLab library, which provides unified state representations, action spaces, and a ReAct-style base agent implementation. Using the base agent and Codex, we perform rejection sampling with GPT-5.2 and GPT-5-mini as the LLMs. The final pool contains 599 trajectories from WebArena and 941 from WorkArena/WorkArena++. The overall success rate is 52.0%, and each trajectory contains 28.1 states on average.

轨迹收集。 我们从 WebArena、WorkArena 和 WorkArena++ 三个网络智能体基准中收集轨迹，使用其中的 OneStopShop、CMS、Reddit 和 ServiceNow 环境。 轨迹通过 AgentLab 库收集，该库提供统一的状态表示、动作空间和 ReAct 风格的基础智能体实现。 我们使用基础智能体和 Codex，以 GPT-5.2 和 GPT-5-mini 作为 LLM 执行拒绝采样。 最终轨迹池包含 599 条 WebArena 轨迹，以及 941 条 WorkArena/WorkArena++ 轨迹。 总体成功率为 52.0%，每条轨迹平均包含 28.1 个状态。

图2：LME-V2 问题覆盖不同的来源领域、问题类型和问题格式。

Question Annotation. All questions are constructed through manual annotation. Following the memory ability taxonomy, human experts first inspect the trajectories to identify various information an experienced colleague would naturally learn. We then curate and filter questions to ensure strong proprietary LLMs cannot answer from parametric knowledge alone. We manually tested Gemini-3-Pro, GPT-5.2, Grok-4.1-thinking, and Claude-Opus-4.6 and ensured that at least two out of four models answered the questions incorrectly.

问题标注。 所有问题都通过人工标注构建。 遵循记忆能力分类体系，人类专家首先检查轨迹，识别经验丰富的同事会自然学到的各类信息。 随后，我们整理并筛选问题，确保强大的闭源 LLM 无法仅凭参数知识作答。 我们人工测试了 Gemini-3-Pro、GPT-5.2、Grok-4.1-thinking 和 Claude-Opus-4.6，并确保四个模型中至少有两个会答错问题。

Gotchas questions are framed as scenarios where an inexperienced worker sends a message with a screenshot, while the other questions are expressed as text-only true/false, multiple choice, or short answer questions. Finally, based on existing static, dynamic, and workflow questions, we curate abstention questions with wrong premises that the model must identify to succeed. Figure 1 shows example questions in each category. Figure 2 presents source domain, type, and format distribution of the final question pool. On average, questions require 1.4 trajectories to answer (min 1, max 5). However, many dynamic and workflow questions require evidence synthesized from many states within a supporting trajectory.

环境陷阱问题被表述为缺乏经验的工作人员发送带有截图的消息，而其他问题则采用纯文本判断题、选择题或简答题形式。 最后，我们基于现有的静态、动态和工作流问题整理带有错误前提的拒答问题，模型必须识别这些错误前提才能成功。 图1展示了各类别的问题示例。 图2展示了最终问题池的来源领域、类型和格式分布。 平均而言，每个问题需要 1.4 条轨迹才能回答，最少 1 条，最多 5 条。 不过，许多动态和工作流问题需要综合支持轨迹中多个状态的证据。

Answer Trajectory Labeling. During annotation, annotators identify a seed set of answer-bearing trajectories for each question. To construct shared history haystacks where we can jointly minimize the number of answer-bearing trajectories for all questions, we perform additional annotation to label all trajectories that contain the answer for each question. We use the Codex coding agent to generate initial proposals. Human experts then verify that the question-trajectory correspondence for trajectories included in the final core haystack set. We provide details in the appendix.

答案轨迹标记。 标注过程中，标注者为每个问题识别一组初始的答案承载轨迹。 为了构建共享历史干草堆，并共同最小化所有问题对应的答案承载轨迹数量，我们进行额外标注，标记每个问题所有包含答案的轨迹。 我们使用 Codex 编码智能体生成初始候选。 随后，人类专家验证最终核心干草堆集合所含轨迹与问题之间的对应关系。 具体细节见附录。

图3：LME-V2 历史干草堆的基本统计。小型和中型层级覆盖大量状态和 token（左）；与此同时，每个问题包含答案的轨迹在干草堆中通常仍然稀疏（右）。

Haystack Creation. Based on the answer trajectory labels, we programmatically assemble two tiers of history trajectory haystacks: a small variant that contains 100 trajectories shared by all questions, and a medium variant that contains roughly 500 trajectories per question. We refer to them as LME-V2-Small and LME-V2-Medium for the rest of the paper. For LME-V2-Small, we create one haystack for the ServiceNow questions and one haystack for the WebArena domains. All haystacks contain a balanced ratio of successful and failed trajectories, and many questions can only be answered from failed trajectories.

干草堆构建。 根据答案轨迹标签，我们以程序方式组装两个层级的历史轨迹干草堆：小型变体包含由所有问题共享的 100 条轨迹，中型变体则为每个问题包含约 500 条轨迹。 在本文其余部分，我们把它们称为 LME-V2-Small 和 LME-V2-Medium。 对于 LME-V2-Small，我们为 ServiceNow 问题创建一个干草堆，并为 WebArena 领域创建另一个干草堆。 所有干草堆都包含比例均衡的成功与失败轨迹，而且许多问题只能从失败轨迹中找到答案。

Figure 3 presents further statistics of the haystacks. The final history lengths of LME-V2-Small and LME-V2-Medium are approximately 25M and 115M tokens, while each question's answer-bearing trajectory set remains sparse in the haystack. Table 1 compares LME-V2 with previous long-term memory benchmarks. LME-V2 has substantially longer histories than prior long-term memory benchmarks, naturally includes multimodal evaluation, and provides a broad coverage of crucial agent memory capabilities.

图3给出了干草堆的更多统计信息。 LME-V2-Small 和 LME-V2-Medium 的最终历史长度分别约为 2500 万和 1.15 亿 token，而每个问题的答案承载轨迹集合在干草堆中仍然稀疏。 表1将 LME-V2 与先前的长期记忆基准进行比较。 LME-V2 的历史比先前长期记忆基准长得多，天然包含多模态评估，并广泛覆盖关键的智能体记忆能力。

3.3 Evaluation Formulation

We formulate LME-V2 as a context gathering task. For each question $q_i$ with gold answer $y_i$, a memory system receives an ordered trajectory haystack ${\mathcal{H}}_i=\{h_{i,1},\dots ,h_{i,m_i}\}$, where each $h_{i,j}$ is a trajectory. The system must support two APIs, $\mathtt{\text{Insert}}(h)$ and $\mathtt{\text{Query}}(q)$. We sequentially insert all trajectories in ${\mathcal{H}}_i$, query the final memory with $q_i$, and obtain a returned context $c_i$:

我们把 LME-V2 形式化为一项上下文收集任务。 对于每个问题 $q_i$ 及其标准答案 $y_i$，记忆系统接收一个有序轨迹干草堆 ${\mathcal{H}}_i=\{h_{i,1},\dots ,h_{i,m_i}\}$，其中每个 $h_{i,j}$ 都是一条轨迹。 系统必须支持两个 API：$\mathtt{\text{Insert}}(h)$ 和 $\mathtt{\text{Query}}(q)$。 我们依次插入 ${\mathcal{H}}_i$ 中的所有轨迹，再使用 $q_i$ 查询最终记忆，并得到返回的上下文 $c_i$：

$$\begin{array}{rl}{\mathcal{M}}_{i,j}& ={\mathtt{\text{Insert}}}_{{\mathcal{M}}_{i,j-1}}(h_{i,j}),\\ c_i& ={\mathtt{\text{Query}}}_{{\mathcal{M}}_{i,m_i}}(q_i).\end{array}$$

A fixed reader model $R$ answers from the question and a bounded memory context: ${\hat{y}}_i=R(q_i,\mathrm{Trunc}(c_i))$. We set the truncation budget to 200k tokens empirically. We report answer accuracy and query latency. Accuracy is computed by normalized string matching for structured answers and an LLM judge for free-form answers.

固定的阅读器模型 $R$ 根据问题和有界记忆上下文作答：${\hat{y}}_i=R(q_i,\mathrm{Trunc}(c_i))$。 我们根据经验把截断预算设为 200k token。 我们报告答案准确率和查询延迟。 对于结构化答案，准确率通过归一化字符串匹配计算；对于自由形式答案，则使用 LLM 裁判计算。

3.4 Pilot Studies

(a) No-context direct QA

Context	Overall	Static	Dynamic	Workflow	Gotchas
Qwen3.5-9B (thinking enabled)
No context	0.016	0.000	0.015	0.155	0.136
Oracle trajectories	0.596	0.566	0.668	0.718	0.310
Oracle slices + notes	0.825	0.908	0.879	0.750	0.484
GPT-5.4-mini (medium reasoning)
No context	0.045	0.025	0.010	0.075	0.171
Oracle trajectories	0.653	0.660	0.696	0.697	0.484
Oracle slices + notes	0.863	0.950	0.843	0.905	0.467
Codex + GPT-5.4-mini (xhigh reasoning)
Oracle trajectory files	0.897	0.986	0.947	0.815	0.517

(b) Oracle-context direct QA

图4：LME-V2 非拒答问题上的先导实验。(a) 在没有轨迹历史时，前沿 LLM 表现很差，说明仅靠参数知识不足以回答 LME-V2。(b) 即使提供包含答案的预言轨迹，LME-V2 仍然难以回答；带笔记的证据切片和编码智能体框架等优化能够提高表现。

We perform two pilot studies. First, we evaluate whether LME-V2 questions require environment-specific trajectory evidence. Then, we sanity check whether answer-bearing trajectories are sufficient for reliable question answering. These studies use a direct question answering setup rather than the context gathering formulation used in the main experiments, and evaluate non-abstention questions only. Full per-category results, prompts, and sandbox instructions are provided in the appendix.

我们开展了两项先导实验。 首先，我们评估 LME-V2 问题是否需要特定环境的轨迹证据。 随后，我们进行合理性检查，确认答案承载轨迹是否足以支持可靠问答。 这些实验采用直接问答设置，而不是主实验使用的上下文收集形式，并且只评估非拒答问题。 完整的分类结果、提示和沙箱说明见附录。

To begin with, can recent frontier LLMs answer LME-V2 questions without the trajectory history? We prompt strong LLMs with only the question. As shown in Figure 4 (left), all LLMs perform poorly in this setting: the best model reaches only 14.1% overall accuracy, suggesting that LME-V2 questions generally cannot be answered from public or parametric knowledge alone.

首先，近期前沿 LLM 能否在没有轨迹历史的情况下回答 LME-V2 问题？ 我们只向强大 LLM 提供问题。 如图4左侧所示，所有 LLM 在这一设置下都表现很差：最佳模型的总体准确率也只有 14.1%，说明通常无法仅依靠公开知识或参数知识回答 LME-V2 问题。

Second, we give models oracle access to the answer-bearing trajectories to isolate the difficulty of reading and grounding trajectory evidence. Long-context prompting shows much higher accuracy but remains limited due to the trajectory size exceeding the model's context window. We further consider two techniques: 1) annotating ground-truth states containing the evidence and providing only radius-1 evidence slices around them and 2) summarizing strategy notes containing important procedures and gotchas identified in the trajectory. These two techniques further improve direct QA to 82.5% and 86.3%, respectively.

其次，我们为模型提供对答案承载轨迹的预言访问，以隔离阅读和落地轨迹证据本身的难度。 长上下文提示取得高得多的准确率，但由于轨迹规模超过模型上下文窗口，表现仍然受限。 我们进一步考虑两项技术：1）标注包含证据的标准状态，只提供其周围半径为 1 的证据切片；2）总结策略笔记，其中包含轨迹中识别出的重要流程和陷阱。 这两项技术分别把直接问答准确率进一步提高到 82.5% 和 86.3%。

Finally, we represent the trajectories as files and use the off-the-shelf Codex coding agent to directly answer the question. Surprisingly, GPT-5.4-mini with the Codex harness answers the questions better than prompting approach, suggesting that detailed evidence inspection via multi-step tool use is effective for understanding agent trajectories, and that coding agents might have good performance acting as memory controllers. Overall, these findings confirm that the answer trajectory labelings are accurate enough and motivate our memory method design.

最后，我们把轨迹表示为文件，并使用开箱即用的 Codex 编码智能体直接回答问题。 令人意外的是，搭配 Codex 框架的 GPT-5.4-mini 比提示方法回答得更好，这说明通过多步工具使用详细检查证据有助于理解智能体轨迹，而且编码智能体作为记忆控制器可能具有良好表现。 总体而言，这些发现确认答案轨迹标签足够准确，并为我们的记忆方法设计提供了动机。

4. AgentRunbook

LME-V2 is challenging because the evidence needed for a question can mix low-level UI observations, state transitions, and reusable task procedures. Memory modules therefore need to organize noisy agent trajectories into compact representations and index them for targeted recall. We propose two memory designs: AgentRunbook-R, a structured RAG pipeline with separate knowledge pools, and AgentRunbook-C, a coding agent based method that casts memorizing agentic contexts as a file management problem. Figure 5 illustrates the workflow of both methods.

LME-V2 具有挑战性，因为回答问题所需的证据可能混合低层 UI 观察、状态转换和可复用任务流程。 因此，记忆模块需要把嘈杂的智能体轨迹组织为紧凑表示，并建立索引以支持定向回忆。 我们提出两种记忆设计：AgentRunbook-R 是具有独立知识池的结构化 RAG 流程；AgentRunbook-C 是基于编码智能体的方法，把智能体上下文的记忆转化为文件管理问题。 图5展示了两种方法的工作流。

图5：AgentRunbook 记忆模块示意图。(a) AgentRunbook-R 在插入时消化轨迹，分别为原始状态、事件和笔记创建专用知识池；查询时，LLM 控制器为每个知识池生成多路查询。(b) AgentRunbook-C 在插入时把轨迹存为文件，并为编码智能体构建包含专用指令和清单的沙箱工作区，以高效收集所需证据。

4.1 AgentRunbook-R

AgentRunbook-R, where R denotes RAG, extracts structured memory items at insertion time and retrieves them at query time. To recall information at different granularities, AgentRunbook-R uses separate knowledge pools and a retrieval mechanism over these pools. Given a trajectory $h$, AgentRunbook-R builds three memory pools. The raw state slice pool stores windows centered at trajectory states, including local UI observations and nearby actions. This pool preserves fine-grained visual and textual evidence.

AgentRunbook-R 中的 R 表示 RAG；它在插入时抽取结构化记忆项，并在查询时检索这些记忆项。 为了回忆不同粒度的信息，AgentRunbook-R 使用独立知识池，并在这些知识池上执行检索。 给定一条轨迹 $h$，AgentRunbook-R 构建三个记忆池。 原始状态切片池存储以轨迹状态为中心的窗口，其中包括局部 UI 观察和邻近动作。 该知识池保留细粒度视觉与文本证据。

The state transition event pool stores events extracted from consecutive states. These events describe how actions change the environment, accumulating evidence for an environment world model. The procedure and hint note pool stores trajectory-level notes that capture reusable workflows, navigation patterns, and environment-specific gotchas. This pool is inspired by prior works that consolidate trajectory experience into compact reusable knowledge.

状态转换事件池存储从连续状态中抽取的事件。 这些事件描述动作如何改变环境，为环境世界模型积累证据。 流程与提示笔记池存储轨迹级笔记，用于捕获可复用工作流、导航模式和特定环境陷阱。 该知识池的设计受先前将轨迹经验整合为紧凑可复用知识的工作启发。

At query time, AgentRunbook-R uses an LLM controller to reason about the query and the current memory snapshot, then generate retrieval queries for the knowledge pools: multiple raw state queries for exact UI evidence, one event query for important state changes, and one note query for procedural knowledge. The controller may skip irrelevant streams. Each query retrieves from its corresponding pool using dense retrieval, and the results are rendered as a multimodal memory context $c_i$. This design keeps query latency low while supporting evidence at different granularities.

查询时，AgentRunbook-R 使用 LLM 控制器对查询和当前记忆快照进行推理，然后为各知识池生成检索查询：为精确 UI 证据生成多个原始状态查询，为重要状态变化生成一个事件查询，并为流程知识生成一个笔记查询。 控制器可以跳过不相关的数据流。 每个查询都使用稠密检索从对应知识池中检索，结果被渲染为多模态记忆上下文 $c_i$。 这一设计在支持不同粒度证据的同时保持较低查询延迟。

4.2 AgentRunbook-C

AgentRunbook-C, where C denotes coding agent, is motivated by the observation that general coding agents are effective file system manipulators and tool users. Rather than compressing retrieval behavior into a fixed vector search pipeline, AgentRunbook-C stores trajectories directly as files and uses a coding agent to search, inspect, and select evidence at query time. An off-the-shelf coding agent, however, is not optimized as a memory module. It may over-explore, under-explore, or inspect the data inefficiently.

AgentRunbook-C 中的 C 表示编码智能体；其动机来自通用编码智能体能够有效操作文件系统和使用工具这一观察。 AgentRunbook-C 不把检索行为压缩进固定的向量搜索流程，而是直接把轨迹存为文件，并在查询时使用编码智能体搜索、检查和选择证据。 然而，开箱即用的编码智能体并未针对充当记忆模块进行优化。 它可能探索过度、探索不足，或者低效地检查数据。

AgentRunbook-C adds three lightweight scaffolding components to the coding agent. First, a workflow document instructs the agent to act as a memory module and outlines the steps for collecting evidence. Second, query-time manifest artifacts summarize the current memory layout, helping the agent shortlist relevant trajectories before detailed inspection. Third, a helper script exposes common trajectory inspection operations, such as viewing a state span or searching within a trajectory.

AgentRunbook-C 为编码智能体增加三个轻量级脚手架组件。 第一，工作流文档指示智能体充当记忆模块，并概述收集证据的步骤。 第二，查询时的清单工件总结当前记忆布局，帮助智能体在详细检查前筛选相关轨迹。 第三，辅助脚本提供常见轨迹检查操作，例如查看某段状态或在轨迹中搜索。

At insertion time, AgentRunbook-C stores each trajectory on disk. At query time, it creates a sandbox with the question, workflow document, helper script, and rendered manifest artifacts. The coding agent then writes a structured retrieval output containing a short memory note and selected trajectory state spans, which are rendered into $c_i$.

插入时，AgentRunbook-C 把每条轨迹存储到磁盘。 查询时，它创建一个包含问题、工作流文档、辅助脚本和已渲染清单工件的沙箱。 随后，编码智能体写出结构化检索结果，其中包含简短记忆笔记和选定的轨迹状态片段，再将其渲染为 $c_i$。

5. Experiments

We evaluate all methods on LME-V2 under the context gathering formulation. The returned memory context is truncated to 200K tokens and answered by a fixed Qwen3.5-9B reader. For RAG methods, we use Qwen3.5-9B as the memory controller and Qwen3-Embedding-8B for retrieval. For coding agent methods, we use Codex and GPT-5.4-mini with different reasoning efforts. The full implementation details of AgentRunbook and baselines are provided in the appendix.

我们在上下文收集形式下评估 LME-V2 上的所有方法。 返回的记忆上下文被截断到 200K token，并由固定的 Qwen3.5-9B 阅读器作答。 对于 RAG 方法，我们使用 Qwen3.5-9B 作为记忆控制器，并使用 Qwen3-Embedding-8B 进行检索。 对于编码智能体方法，我们使用具有不同推理强度的 Codex 和 GPT-5.4-mini。 AgentRunbook 和基线的完整实现细节见附录。

5.1 Main Results

表2：基线与消融的主结果。下游阅读器始终为 Qwen3.5-9B；每个方法族中的最佳结果以粗体表示。* 表示通过配对 bootstrap 检验，在统计上显著优于非消融基线（p < 0.05）。AgentRunbook 在 RAG 方法族中显著优于基线，并在编码智能体方法族中取得更优延迟。

Method	LME-V2-Small						LME-V2-Medium
	Overall	Static	Dynamic	Workflow	Gotchas	Latency	Overall	Static	Dynamic	Workflow	Gotchas	Latency
No retrieval	0.013	0.000	0.008	0.094	0.138	0s	0.013	0.000	0.008	0.094	0.138	0s
RAG Methods (Controller = Qwen3.5-9B, thinking enabled)
RAG: query → slice	0.428	0.471	0.425	0.415	0.207	0.1s	0.381	0.434	0.405	0.293	0.242	0.1s
RAG: query → slice + notes	0.510	0.524	0.496	0.528	0.414	0.2s	0.459	0.487	0.472	0.434	0.310	0.3s
AgentRunbook-R	0.586*	0.661*	0.583*	0.528	0.310	26.9s	0.570*	0.630*	0.614*	0.472*	0.345	25.8s
− raw slice pool	0.423	0.286	0.551	0.538	0.345	16.7s	0.335	0.233	0.433	0.377	0.413	17.1s
− event pool	0.556	0.614	0.559	0.528	0.276	19.1s	0.484	0.534	0.496	0.434	0.276	18.5s
− note pool	0.579	0.651	0.614	0.481	0.310	22.8s	0.499	0.561	0.543	0.396	0.276	20.5s
Coding Agent Methods (Controller = GPT-5.4-mini, xhigh reasoning)
Codex	0.699	0.804	0.670	0.575	0.586	177.2s	0.687	0.783	0.646	0.613	0.517	185.8s
AgentRunbook-C	0.749*	0.820*	0.724*	0.726*	0.483	108.3s	0.701	0.788	0.701*	0.613	0.449	139.9s
− workflow	0.701	0.772	0.677	0.632	0.586	167.9s	0.641	0.709	0.646	0.575	0.414	231.9s
− manifest artifacts	0.747	0.847	0.709	0.698	0.448	155.0s	0.681	0.767	0.685	0.576	0.483	211.6s
− helper functions	0.714	0.783	0.724	0.660	0.414	145.9s	0.718	0.804	0.693	0.689	0.380	182.5s

As shown in Table 2, the no-retrieval baseline is near zero, confirming that the reader cannot answer without memory context. A simple query-to-slice RAG baseline reaches 42.8% on LME-V2-Small and 38.1% on LME-V2-Medium, while adding trajectory notes improves performance to 51.0% and 45.9%. AgentRunbook-R further improves over the strongest RAG baseline, reaching 58.6% on LME-V2-Small and 57.0% on LME-V2-Medium. The ablations show that the raw slice pool is important for static questions, while removing the event pool harms static, dynamic, and gotchas questions. Workflow questions benefit from consolidating trajectory experience into reusable events and notes rather than only retrieving local observations.

如表2所示，无检索基线接近于零，确认阅读器在没有记忆上下文时无法回答问题。 简单的查询到切片 RAG 基线在 LME-V2-Small 和 LME-V2-Medium 上分别达到 42.8% 和 38.1%，加入轨迹笔记后则提高到 51.0% 和 45.9%。 AgentRunbook-R 进一步超过最强 RAG 基线，在 LME-V2-Small 和 LME-V2-Medium 上分别达到 58.6% 和 57.0%。 消融实验表明，原始切片池对静态问题十分重要，而移除事件池会损害静态、动态和环境陷阱问题的表现。 对于工作流问题，把轨迹经验整合为可复用事件和笔记，比只检索局部观察更有帮助。

AgentRunbook-C achieves the best overall accuracy, reaching 74.9% on LME-V2-Small and 70.1% on LME-V2-Medium. It also improves over vanilla Codex, which reaches 69.9% and 68.7%, respectively. The ablations show that workflow instructions are consistently important, while manifest artifacts mainly improve efficiency. Helper functions affect the performance in a mixed way: they improve the small-tier result and reduce latency compared with the most expensive ablations, but their effect on medium-tier accuracy is not uniformly positive. Overall, the results show that coding agents can serve as strong memory controllers with a proper file-based environment. In the appendix, we further analyze the error patterns and the tool calling behavior of AgentRunbook-C.

AgentRunbook-C 取得最高的总体准确率，在 LME-V2-Small 和 LME-V2-Medium 上分别达到 74.9% 和 70.1%。 它也超过原始 Codex，后者分别达到 69.9% 和 68.7%。 消融实验表明，工作流指令始终十分重要，而清单工件主要提高效率。 辅助函数对性能的影响并不一致：与开销最大的消融设置相比，它们提高小型层级结果并降低延迟，但对中型层级准确率的影响并非始终为正。 总体而言，结果表明，在适当的文件式环境中，编码智能体可以充当强大的记忆控制器。 我们在附录中进一步分析 AgentRunbook-C 的错误模式和工具调用行为。

5.2 Accuracy and Latency Trade-off

图6：AgentRunbook 推进了记忆模块的查询准确率-延迟前沿。

An ideal memory method should support both accurate and efficient querying. Across multiple factors, we find the reasoning effort of the memory controller has a large and direct effects on the overall query latency. We thus use it to analyze the methods across different operating points. As shown in Figure 6, AgentRunbook-R provides a moderate-accuracy, low-latency baseline: it substantially improves over the slice-plus-note baseline while keeping latency around 26 seconds, or much lower without thinking.

理想的记忆方法应同时支持准确且高效的查询。 在多个因素中，我们发现记忆控制器的推理强度会对总体查询延迟产生显著而直接的影响。 因此，我们用它分析不同运行点上的方法。 如图6所示，AgentRunbook-R 提供中等准确率、低延迟的基线：它在显著超过切片加笔记基线的同时，把延迟保持在约 26 秒；关闭思考后延迟还会低得多。

This makes it a strong choice when query efficiency is prioritized. AgentRunbook-C moves the accuracy and latency frontier upward. Across reasoning effort settings, the scaffolded coding agent memory consistently offers a better trade-off than directly using the off-the-shelf coding agent. This suggests that coding agents are more effective as memory controllers when paired with explicit workflow guidance, manifests, and trajectory inspection tools.

当查询效率优先时，这使它成为有力选择。 AgentRunbook-C 则把准确率和延迟前沿向上推进。 在不同推理强度设置下，带脚手架的编码智能体记忆始终比直接使用开箱即用的编码智能体提供更好的权衡。 这说明，当编码智能体搭配显式工作流指导、清单和轨迹检查工具时，能够更有效地充当记忆控制器。

6. Conclusion

We introduce LongMemEval-V2, a long-term memory benchmark that formulates a new standard for agent memory evaluation: memory systems should help agents become experienced operators of specialized environments. LME-V2 holistically covers five memory abilities and advances the context depth of memory benchmarks with beyond 100M-token context from large multimodal web-agent histories. We further propose AgentRunbook-R, which improves standard RAG-based methods with dedicated memory pools, and AgentRunbook-C, which leverages the file manipulation abilities of coding agents and further improves the accuracy and latency through lightweight workflow guidance, manifests, and inspection tools. We hope LME-V2 provides a concrete testbed for memory modules that make long-running agents more intelligent and reliable in real-world environments.

我们提出 LongMemEval-V2，这是一项长期记忆基准，为智能体记忆评估制定了新标准：记忆系统应帮助智能体成为专用环境中经验丰富的操作人员。 LME-V2 整体覆盖五项记忆能力，并通过来自大规模多模态网络智能体历史、超过 1 亿 token 的上下文推进记忆基准的上下文深度。 我们还提出 AgentRunbook-R，它通过专用记忆池改进标准 RAG 方法；以及 AgentRunbook-C，它利用编码智能体的文件操作能力，并通过轻量级工作流指导、清单和检查工具进一步改善准确率与延迟。 我们希望 LME-V2 为记忆模块提供具体试验平台，使长期运行的智能体在真实环境中更智能、更可靠。