---

Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory

MemoryAgentRL埃森哲

Wang Zhenting, Chen Huancheng, Wang Jiayun, et al. Memex(RL): scaling long-horizon LLM agents via indexed experience memory[PP/OL]. V1. arXiv (2026-03-04)[2026-10-23]. DOI:10.48550/arXiv.2603.04257.

https://arxiv.org/abs/2603.04257v1

---

Memex(RL)：通过索引经验记忆扩展长时程 LLM 智能体

Abstract

Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence.

大语言模型（LLM）智能体在长时程任务中受到有限上下文窗口的根本制约。 随着轨迹增长，将工具输出和中间推理保留在上下文中很快就变得不可行：工作上下文会变得过长，最终超出上下文预算，而且即使较早的证据仍然存在，也会更难被利用。 现有方案通常通过截断或滚动摘要来缩短上下文，但这些方法从根本上是有损的，因为它们压缩或丢弃了过去的证据本身。 我们提出 Memex，这是一种索引经验记忆机制，能够在不丢弃证据的情况下压缩上下文。

Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.

Memex 维护由简洁的结构化摘要和稳定索引组成的紧凑工作上下文，同时以这些索引为键，将底层交互完整保真地存入外部经验数据库。 随后，智能体可以决定何时解引用某个索引，恢复当前子目标所需的确切历史证据。 我们使用强化学习框架 MemexRL 同时优化写入与读取行为，并针对上下文预算约束下的索引记忆使用设计奖励塑形，使智能体学会摘要什么、归档什么、如何建立索引，以及何时检索。 与仅依赖摘要的方法相比，这提供了一种信息损失显著更少的长时程记忆形式。 我们还给出理论分析，说明 Memex 循环有望通过有界次数的解引用保持决策质量，同时在历史不断增长时，让有效的上下文内计算保持有界。 实验表明，在具有挑战性的长时程任务上，经过 MemexRL 训练的 Memex 智能体提高了任务成功率，同时使用显著更小的工作上下文。

1. Introduction

Large language model (LLM) agents are increasingly deployed as general-purpose problem solvers that plan, call tools, and interact with users over extended periods. Unlike single-shot prompting or short multi-turn chats, long-horizon agents are asked to execute workflows spanning dozens to hundreds of steps and tool calls, often producing large volumes of intermediate observations, tool outputs, and reasoning traces along the way, such as searching and cross-referencing scientific literature, exploring configuration spaces for code and infrastructure, orchestrating multi-API business processes, or iteratively refining complex analyses for a user. In these settings, success depends not only on local reasoning quality, but also on whether the agent can preserve and later reuse information that first appeared many steps earlier, such as a constraint mentioned at the beginning of a conversation, a failure mode surfaced by a tool, or an API response that only becomes decisive much later.

大语言模型（LLM）智能体正越来越多地被部署为通用问题求解器，能够在较长时间内进行规划、调用工具并与用户交互。 与单次提示或简短的多轮对话不同，长时程智能体需要执行包含数十到数百个步骤和工具调用的工作流，期间通常会产生大量中间观测、工具输出和推理轨迹，例如检索并交叉核对科学文献、探索代码与基础设施的配置空间、编排涉及多个 API 的业务流程，或为用户迭代完善复杂分析。 在这些场景中，成功不仅取决于局部推理质量，还取决于智能体能否保留并在之后复用许多步骤之前首次出现的信息，例如对话开头提到的约束、工具暴露出的故障模式，或直到很久以后才起决定性作用的 API 响应。

This creates a fundamental bottleneck for long-horizon agents. Although modern LLMs support increasingly large context windows, those windows remain finite, while agent trajectories naturally keep growing as observations, tool outputs, and intermediate reasoning are appended over time. A straightforward strategy is to retain as much of this history as possible in-context, but this quickly becomes inefficient or infeasible. Prompts become prohibitively long, eventually exceed the available context budget, and make distant evidence harder to use even when it is still present.

这为长时程智能体带来了一个根本瓶颈。 尽管现代 LLM 支持的上下文窗口越来越大，但窗口仍然有限；与此同时，随着观测、工具输出和中间推理不断追加，智能体轨迹会自然地持续增长。 一种直接的策略是尽可能多地将这些历史保留在上下文中，但这很快就会变得低效或不可行。 提示会变得过长，最终超出可用的上下文预算，而且即使较早的证据仍然存在，也会更难被利用。

Existing systems mostly address this pressure through static context engineering, such as large rolling prompts, heuristic summarization, or related memory heuristics. These approaches can reduce active working context, but they usually do so by either truncating substantial portions of past interactions or compressing them into lossy summaries that are difficult to faithfully recover later. A natural alternative is to log everything into an external memory and retrieve past content by semantic similarity when needed. However, in long-horizon tool use, this design is often brittle.

现有系统主要通过静态上下文工程来缓解这种压力，例如大型滚动提示、启发式摘要或相关的记忆启发式方法。 这些方法可以缩减活跃工作上下文，但通常是通过截断大量历史交互，或将其压缩成之后难以忠实恢复的有损摘要来实现的。 一种自然的替代方案是将所有内容记录到外部记忆中，并在需要时通过语义相似度检索过去的内容。 然而，在长时程工具使用中，这种设计往往很脆弱。

When memory consists of a large pool of noisy, near-duplicate fragments, retrieval becomes ambiguous, and the model must repeatedly re-parse loosely structured history. More fundamentally, similarity-based retrieval does not specify how the agent should organize its own experience. It does not determine which intermediate results deserve stable references, which branches are dead ends, or how artifacts should be named so that later access is precise rather than fuzzy. As a result, many current systems still rely on hand-designed templates and heuristics for memory construction and retrieval. This leaves a gap between how humans manage long-term work, keeping a small set of active concepts while preserving exact notes and references externally, and how LLM agents manage theirs.

当记忆由大量嘈杂、近乎重复的片段组成时，检索会变得模糊，模型必须反复重新解析结构松散的历史。 更根本的问题是，基于相似度的检索并未规定智能体应如何组织自身经验。 它无法决定哪些中间结果值得拥有稳定引用、哪些分支已经走入死胡同，也无法决定如何命名各类产物，才能让后续访问精确而非模糊。 因此，许多现有系统仍依赖人工设计的模板和启发式规则来构建与检索记忆。 这使 LLM 智能体管理长期工作的方式与人类之间仍有差距：人类只让少量概念保持活跃，同时在外部保存精确的笔记和引用。

图1：Memex 智能体循环概览。CompressExperience 将上下文中的长工具使用轨迹替换为紧凑的索引摘要，同时把详细内容存入外部键值存储。随后，ReadExperience(index) 通过解引用索引取回确切内容，并将其重新注入上下文，从而在较小的上下文窗口内实现长时程执行。

In this work, we argue that long-horizon LLM agents need a memory mechanism that compresses context without discarding evidence. We instantiate this idea with Memex, whose core component is Indexed Experience Memory. As illustrated in Figure 1, Memex replaces a long tool-use trajectory in the working context with a compact indexed summary, while archiving the full underlying artifacts in an external key–value experience store under stable indices. Later, when a specific past result becomes relevant, the agent can explicitly dereference an index to recover the exact archived content and re-inject it into the working context.

在本文中，我们认为，长时程 LLM 智能体需要一种 在不丢弃证据的情况下压缩上下文 的记忆机制。 我们以 Memex 实现这一思想，其核心组件是 索引经验记忆。 如 图1 所示，Memex 用紧凑的索引摘要替换工作上下文中的长工具使用轨迹，同时以稳定索引将完整的底层产物归档到外部键值经验存储中。 随后，当某个过去的结果变得相关时，智能体可以显式解引用一个索引，恢复确切的归档内容，并将其重新注入工作上下文。

In this way, Memex explicitly separates a compact in-context working context from a full-fidelity external experience archive. At each step, the agent writes a concise, structured indexed summary into its working context, recording actionable progress together with stable references. In parallel, the full underlying artifacts needed for faithful later reuse, such as tool outputs, logs, code snippets, and other fine-grained evidence, are archived in the external store under those indices. Subsequent decisions operate primarily over this short indexed working context, and the agent brings back raw evidence only through explicit index dereferencing when it becomes relevant to the current subgoal.

通过这种方式，Memex 将紧凑的上下文内 工作上下文 与完整保真的外部 经验档案 显式分离。 每一步中，智能体都会向工作上下文写入简洁、结构化的 索引摘要，记录可据以行动的进展以及稳定引用。 与此同时，为供日后忠实复用所需的完整底层产物，例如工具输出、日志、代码片段和其他细粒度证据，会以这些索引为键归档到外部存储中。 后续决策主要基于这一简短的索引工作上下文；只有当原始证据与当前子目标相关时，智能体才会通过显式索引解引用将其取回。

This makes memory access precise and auditable because an index points to a concrete archived artifact rather than to an approximate semantic match. Crucially, Memex is not a hand-crafted memory heuristic. We treat memory operations, including writing indexed summaries, archiving artifacts, and dereferencing indices, as first-class actions in the same decision space as environment tools. This introduces a distinctive long-horizon credit assignment problem.

这使记忆访问既精确又可审计，因为索引指向的是具体的归档产物，而非近似的语义匹配结果。 关键在于，Memex 并不是一种人工构造的记忆启发式方法。 我们将记忆操作，包括写入索引摘要、归档产物和解引用索引，视为与环境工具处于同一决策空间中的一等动作。 这带来了一个独特的长时程信用分配问题。

A well-timed compression or a well-designed index may only pay off many steps later by enabling precise evidence recovery, avoiding redundant tool calls, and preventing context overflow. In contrast, a locally plausible but poorly structured summary can silently derail downstream reasoning. To learn these behaviors, we introduce MemexRL, a reinforcement learning framework that optimizes both the write policy, including what to summarize, what to archive, how to index, and when to compress, and the read policy, including when and what to dereference, under a context budget. MemexRL combines reward shaping tailored to indexed memory usage with a compression-adaptive training procedure that preserves learning signal for delayed memory decisions across long trajectories.

一次时机恰当的压缩或一个设计良好的索引，可能要到许多步骤以后，才会通过精确恢复证据、避免冗余工具调用和防止上下文溢出而产生回报。 相反，一个局部看似合理但结构不佳的摘要，可能在不易察觉的情况下使下游推理偏离正轨。 为学习这些行为，我们提出 MemexRL，这一强化学习框架在上下文预算约束下同时优化 写入 策略，包括摘要什么、归档什么、如何建立索引及何时压缩，以及 读取 策略，包括何时解引用及解引用哪些内容。 MemexRL 将针对索引记忆使用的奖励塑形与适应压缩的训练过程相结合，在长轨迹中为延迟生效的记忆决策保留学习信号。

We also expose context status to the agent through a soft triggering mechanism, turning compression timing into a learnable skill rather than a fixed system rule. Conceptually, Memex follows a simple principle: keep the active reasoning state small, but do not throw evidence away. Instead of forcing the agent to repeatedly reason over an ever-growing prompt or over lossy summaries, Memex maintains a compact control state in-context while preserving exact past interactions off-context for later reuse. This design is closer to how humans manage long, tool-heavy work, where external artifacts such as notes, file names, and bookmarks serve as stable access routes to detailed evidence without requiring everything to remain in working memory.

我们还通过软触发机制向智能体提供上下文状态，将压缩时机转化为一项可学习的技能，而非固定的系统规则。 从概念上看，Memex 遵循一个简单原则：保持活跃推理状态精简，但不丢弃证据。 Memex 不强迫智能体反复基于不断增长的提示或有损摘要进行推理，而是在上下文中维护紧凑的控制状态，并在上下文之外保存确切的历史交互，以供后续复用。 这种设计更接近人类处理长期、密集使用工具的工作的方式：笔记、文件名和书签等外部产物为详细证据提供稳定的访问路径，无需让所有内容都保留在工作记忆中。

It also echoes index-based accounts of episodic memory, in which compact retrieval cues enable later reactivation of richer underlying traces. Memex operationalizes these principles in an agent setting. The indexed summary serves as an actionable working state, while stable indices provide explicit retrieval cues into a full-fidelity experience database.

这也呼应了情景记忆的索引理论：紧凑的检索线索能够在日后重新激活更丰富的底层痕迹。 Memex 将这些原则落实到智能体场景中。 索引摘要充当可据以行动的工作状态，而稳定索引则为完整保真的经验数据库提供显式检索线索。

To clarify why this design is promising, we also provide a theoretical analysis of the Memex loop. We show its potential to support two desirable properties at the same time: preserving decision quality through bounded explicit dereferencing, and keeping the agent's effective in-context computation bounded as the full message history grows. This analysis is not meant to claim that MemexRL always learns such summaries in practice. Rather, it characterizes the regime in which compact indexed summaries together with bounded retrieval are sufficient to support accurate decisions with bounded working context.

为说明这一设计为何具有潜力，我们还对 Memex 循环进行了理论分析。 我们展示了它同时支持两个理想性质的潜力：通过有界次数的显式解引用保持决策质量，并在完整消息历史增长时，让智能体有效的上下文内计算保持有界。 这一分析并不是要声称 MemexRL 在实践中总能学会这样的摘要。 相反，它刻画了这样一种情形：紧凑的索引摘要与有界检索相结合，足以在有界工作上下文下支持准确决策。

We then study whether this theoretical potential can be realized in practice. To do so, we evaluate Memex on challenging long-horizon tasks that require agents to interleave planning and tool calls over dozens to hundreds of steps while revisiting fine-grained evidence long after it first appears. In this regime, Memex improves task success while operating with a small active working context under tight context budgets. Overall, this paper makes four contributions.

随后，我们研究这一理论潜力能否在实践中实现。 为此，我们在具有挑战性的长时程任务上评估 Memex：这些任务要求智能体在数十到数百个步骤中交替进行规划和工具调用，并在细粒度证据首次出现很久以后重新访问它。 在这种情形下，Memex 在严格的上下文预算内，以较小的活跃工作上下文运行，同时提高任务成功率。 总体而言，本文做出四项贡献。

First, we introduce Indexed Experience Memory, a memory interface that pairs a compact in-context indexed summary with a full-fidelity external experience archive under stable references, enabling precise explicit dereferencing of past evidence. Second, we propose MemexRL, a reinforcement learning framework that trains memory write and read behaviors with reward shaping and compression-adaptive trajectory processing. Third, we provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Fourth, we provide an empirical study on long-horizon tasks showing that learned indexed experience memory improves task success under tight context budgets while maintaining a small active working context.

第一，我们提出索引经验记忆，这一记忆接口将紧凑的上下文内索引摘要与具有稳定引用的完整保真外部经验档案配对，使智能体能够对过去的证据进行精确、显式的解引用。 第二，我们提出 MemexRL，这一强化学习框架通过奖励塑形和适应压缩的轨迹处理来训练记忆写入与读取行为。 第三，我们给出理论分析，展示 Memex 循环通过有界次数的解引用保持决策质量，同时在历史增长时让有效的上下文内计算保持有界的潜力。 第四，我们在长时程任务上开展实证研究，表明学得的索引经验记忆能够在严格的上下文预算下提高任务成功率，同时维持较小的活跃工作上下文。

2. Related Work

LLM Memory. LLM memory has emerged as an important research direction for enabling models and agents to store, organize, and retrieve information beyond their immediate context. Early frameworks such as MemGPT and MemoryBank, which organize dialogue history, accumulate experience, and self-evolve through continuous multi-turn updates, have inspired a line of subsequent works that focus on efficiently storing memory using various structured representations by dynamic mechanism. Long-term memory in LLM agents can be broadly categorized into factual memory and experiential memory. Factual memory functions as a declarative knowledge base that supports consistency, coherence, and logs environment states, whereas experiential memory captures knowledge accumulated from historical trajectories and interaction patterns, enabling continual learning and self-evolution. Another line of work integrates information directly into a model’s internal components—such as hidden states and key–value caches, rather than relying on external memory databases, in pursuit of more efficient information compression. Despite these advances, a central challenge in LLM memory remains how to preserve salient information while enabling efficient compression and retrieval.

LLM 记忆。 LLM 记忆已成为一个重要研究方向，旨在使模型和智能体能够存储、组织和检索超出当前上下文的信息。 MemGPT 和 MemoryBank 等早期框架通过持续的多轮更新来组织对话历史、积累经验并实现自我演化，启发了一系列后续工作，关注通过动态机制使用各种结构化表示来高效存储记忆。 LLM 智能体的长期记忆大致可分为事实记忆和经验记忆。 事实记忆充当陈述性知识库，支持一致性与连贯性，并记录环境状态；经验记忆则捕捉从历史轨迹和交互模式中积累的知识，使持续学习和自我演化成为可能。 另一类工作将信息直接整合到模型内部组件中，例如隐藏状态和键值缓存，而不依赖外部记忆数据库，以追求更高效的信息压缩。 尽管取得了这些进展，LLM 记忆仍面临一个核心挑战：如何在保留重要信息的同时实现高效压缩和检索。

Long-Horizon LLM Agents. Although LLMs have finite context windows, long-horizon agent trajectories naturally accumulate observations, actions, and intermediate reasoning over many steps, causing the prompt to grow until it becomes inefficient or exceeds the available context budget. This ever-growing prompt not only increases inference cost and memory usage, but also makes it harder for the model to effectively use distant evidence even when that evidence remains somewhere in the retained history. To address this challenge, recent work has explored learned memory management for long-horizon settings. Early RL-based approaches such as MEM1, MemAgent, Memory-R1, and Mem-$\alpha$ train models to compress, organize, or update historical information into compact memories, primarily in question-answering or long-context settings.

长时程 LLM 智能体。 尽管 LLM 的上下文窗口有限，长时程智能体轨迹会在多个步骤中自然地积累观测、动作和中间推理，导致提示持续增长，直至变得低效或超出可用的上下文预算。 这种不断增长的提示不仅增加了推理成本和内存占用，也让模型更难有效利用较早的证据，即使这些证据仍保留在历史中的某处。 为应对这一挑战，近期工作开始探索适用于长时程场景的学习式记忆管理。 MEM1、MemAgent、Memory-R1 和 Mem-$\alpha$ 等早期基于 RL 的方法，主要在问答或长上下文场景中训练模型将历史信息压缩、组织或更新为紧凑记忆。

More recent work extends learned summarization to interactive agentic reinforcement learning and long-horizon search. SUPO and ReSum incorporate memory summarization into multi-turn tool-use and search pipelines, while FoldGRPO and AgentFold further study structured context folding and multi-scale historical summaries. However, most existing approaches still center on lossy summary compression or generic memory organization: they reduce the active context, but typically do not preserve a precise, full-fidelity archive of past evidence that can be deterministically revisited later. In contrast, Memex formulates memory as indexed experience management: it keeps a compact in-context working state, while archiving full-fidelity artifacts off-context under reusable stable indices. This design allows the agent to not only decide when to compress, but also what to archive, how to index it, and when to explicitly dereference exact past evidence for downstream reasoning, making memory access substantially less lossy than summary-only approaches.

更新的工作将学习式摘要扩展到交互式智能体强化学习和长时程搜索。 SUPO 和 ReSum 将记忆摘要整合到多轮工具使用与搜索流程中，而 FoldGRPO 和 AgentFold 则进一步研究结构化上下文折叠与多尺度历史摘要。 然而，大多数现有方法仍以有损摘要压缩或通用记忆组织为中心：它们缩减了活跃上下文，却通常不会保留一份精确、完整保真的历史证据档案，供日后确定性地重新访问。 相比之下，Memex 将记忆表述为索引经验管理：它在上下文内保持紧凑的工作状态，同时以可复用的稳定索引，将完整保真的产物归档到上下文之外。 这种设计使智能体不仅能决定何时压缩，还能决定归档什么、如何建立索引，以及何时为下游推理显式解引用确切的历史证据，让记忆访问比仅依赖摘要的方法显著减少信息损失。

3. Memex Agent

3.1. Overview

LLM agents must operate under a fixed context window. As interactions accumulate, keeping the entire trajectory in-context quickly becomes infeasible, while lossy compression often removes precisely the evidence needed later (e.g., logs, thinking processes, and intermediate tool outputs). The key component of Memex agent is the Indexed Experience Memory, which keeps a compact in-context state while preserving full-fidelity artifacts in an external experience store under stable indices. The agent periodically rewrites its working context into a short, actionable indexed summary and accesses archived evidence only through explicit index dereferencing when needed.

LLM 智能体必须在固定的上下文窗口下运行。 随着交互积累，将完整轨迹保留在上下文中很快就会变得不可行，而有损压缩往往恰好会移除之后需要的证据，例如日志、思考过程和中间工具输出。 Memex 智能体的关键组件是 索引经验记忆，它在上下文中保持紧凑状态，同时以稳定索引将完整保真的产物保存在外部经验存储中。 智能体定期将工作上下文重写为简短、可据以行动的 索引摘要，并且只在需要时通过显式索引解引用访问归档证据。

3.2. Indexed Experience Memory

We maintain the agent context window $\mathbf{M}$ and an external experience store $\mathcal{D}:\mathsf{index}\mapsto \mathsf{content}$, a key–value database accessed by explicit index dereferencing. Throughout this paper, the working context refers to the dynamic portion of $\mathbf{M}$ beyond the fixed system prompt and task instruction. Namely, $\mathbf{M}=[m_0,u,{\mathbf{M}}_{\mathrm{work}}]$, where ${\mathbf{M}}_{\mathrm{work}}$ denotes the working context. Depending on the stage of interaction, ${\mathbf{M}}_{\mathrm{work}}$ may include the indexed summary, the agent's intermediate reasoning, tool calls, tool outputs, and any retrieved experience blocks injected back into context. We next define the indexed summary $\sigma$ and the indexed experience memory operations.

我们维护智能体上下文窗口 $\mathbf{M}$ 和外部经验存储 $\mathcal{D}:\mathsf{index}\mapsto \mathsf{content}$，后者是通过显式索引解引用访问的键值数据库。 在本文中，工作上下文 指的是 $\mathbf{M}$ 中除固定系统提示和任务指令以外的动态部分。 即 $\mathbf{M}=[m_0,u,{\mathbf{M}}_{\mathrm{work}}]$，其中 ${\mathbf{M}}_{\mathrm{work}}$ 表示工作上下文。 根据交互所处的阶段，${\mathbf{M}}_{\mathrm{work}}$ 可以包括索引摘要、智能体的中间推理、工具调用、工具输出，以及重新注入上下文的检索经验块。 下面定义索引摘要 $\sigma$ 和索引经验记忆操作。

Definition 1 (Indexed Summary). Given an external experience store $\mathcal{D}:\mathsf{index}\mapsto \mathsf{content}$, an Indexed Experience Summary $\sigma$ is an in-context state $\sigma =(s,\mathcal{I})$, where $s$ is a compact, actionable progress state (e.g., verified information, and plans), and $\mathcal{I}$ is a finite set of pairs $\mathcal{I}\triangleq \{(\mathsf{index},\mathsf{description})\},$ where $\mathsf{index}\in \mathrm{dom}(\mathcal{D})$ is a stable index and $\mathsf{description}$ is a summarized descriptor.

定义1（索引摘要）。 给定外部经验存储 $\mathcal{D}:\mathsf{index}\mapsto \mathsf{content}$，索引经验摘要 $\sigma$ 是上下文内状态 $\sigma =(s,\mathcal{I})$，其中 $s$ 是紧凑、可据以行动的进展状态，例如已验证的信息和计划；$\mathcal{I}$ 是由二元组组成的有限集合 $\mathcal{I}\triangleq \{(\mathsf{index},\mathsf{description})\},$ 其中 $\mathsf{index}\in \mathrm{dom}(\mathcal{D})$ 是稳定索引，$\mathsf{description}$ 是摘要式描述。

Definition 2 (Indexed Experience Memory). Let $\mathbf{M}$ denote the agent's context window, initialized as $\mathbf{M}=[m_0,u]$ (system prompt and task instruction), and let $\mathcal{D}$ be an external experience store. An Indexed Experience Memory is a pair $(\mathsf{IndexedSummary},\mathcal{D})$ with:

定义2（索引经验记忆）。 令 $\mathbf{M}$ 表示智能体的上下文窗口，初始化为 $\mathbf{M}=[m_0,u]$（系统提示和任务指令），并令 $\mathcal{D}$ 为外部经验存储。 索引经验记忆 是一个二元组 $(\mathsf{IndexedSummary},\mathcal{D})$，具有以下组成与操作：

(i) In-context indexed summary. $\mathsf{IndexedSummary}$ is a structured state kept in-context that records the actionable progress state, and provides an index map that binds semantic summarized descriptions to indices in $\mathcal{D}$, enabling explicit reference to archived evidence. See Definition 1 for detailed definition.

(i) 上下文内索引摘要。 $\mathsf{IndexedSummary}$ 是保留在上下文中的结构化状态，记录可据以行动的进展状态，并提供 索引映射，将语义摘要描述与 $\mathcal{D}$ 中的索引绑定，从而能够显式引用归档证据。 详细定义见 定义1。

(ii) External Store. $\mathcal{D}:\mathsf{index}\mapsto \mathsf{content}$ maps stable indices to archived content blocks (e.g., tool outputs, traces, code snippets). The agent can access archived evidence only by dereferencing indices via $\mathrm{ReadExperience}(i)$, which returns $\mathcal{D}[i]$.

(ii) 外部存储。 $\mathcal{D}:\mathsf{index}\mapsto \mathsf{content}$ 将 稳定索引 映射到归档内容块，例如工具输出、轨迹和代码片段。 智能体只能通过 $\mathrm{ReadExperience}(i)$ 解引用索引来访问归档证据，该操作返回 $\mathcal{D}[i]$。

(iii) Compression Operation. CompressExperience(IndexedSummary, MemoryBlocks) performs archival by writing each $(\mathsf{index},\mathsf{content})\in \mathsf{MemoryBlocks}$ into $\mathcal{D}$, and then rewrites the working context to $\mathbf{M}\leftarrow [m_0,u,\mathsf{IndexedSummary}]$. Intuitively, it transforms an ever-growing working context into a small indexed summary.

(iii) 压缩操作。 CompressExperience(IndexedSummary, MemoryBlocks) 通过将每个 $(\mathsf{index},\mathsf{content})\in \mathsf{MemoryBlocks}$ 写入 $\mathcal{D}$ 来执行归档，随后将工作上下文重写为 $\mathbf{M}\leftarrow [m_0,u,\mathsf{IndexedSummary}]$。 直观地说，它将不断增长的工作上下文转换为一个小型索引摘要。

(iv) Read Operation. $\mathrm{ReadExperience}(\mathsf{index})$ returns the archived block by dereferencing $o\leftarrow \mathcal{D}[\mathsf{index}]$, and appends it to the working context as a new message $\mathbf{M}\leftarrow \mathbf{M}\oplus [o]$.

(iv) 读取操作。 $\mathrm{ReadExperience}(\mathsf{index})$ 通过解引用 $o\leftarrow \mathcal{D}[\mathsf{index}]$ 返回归档块，并将其作为新消息追加到工作上下文，即 $\mathbf{M}\leftarrow \mathbf{M}\oplus [o]$。

For compression operation, note that the content in each memory block supports two modes: (a) explicit authoring, where the model writes the content directly (e.g., reorganized notes or summarized findings); and (b) anchor-based extraction, where the model specifies three short text anchors (start_anchor, mid_anchor, end_anchor) that uniquely identify a span within the current conversation. The system locates the matching span and archives it verbatim, with the mid_anchor serving as a verification checkpoint to prevent false matches. This dual-mode design gives the model flexibility to paraphrase for storage efficiency or preserve exact content (e.g., precise object IDs, code snippets, test outputs) without generating redundant tokens.

对于压缩操作，需要注意的是，每个记忆块中的内容支持两种模式：(a) 显式编写，模型直接写入内容，例如重新组织的笔记或摘要后的发现；(b) 基于锚点的提取，模型指定三个简短的文本锚点（start_anchor、mid_anchor、end_anchor），以唯一确定当前对话中的一段内容。 系统定位匹配片段并逐字归档，其中 mid_anchor 充当验证检查点，以防止错误匹配。 这种双模式设计使模型可以灵活选择改写内容以提高存储效率，或保留确切内容，例如精确的对象 ID、代码片段和测试输出，而无需生成冗余 token。

Examples for working context compression. We first illustrate how Indexed Experience Memory's compression operation rewrites an ever-growing context into a compact, pointer-heavy state. Here we show a concrete before/after compression example: multiple rounds of reasoning and tool outputs are archived into $\mathcal{D}$ under stable indices, while the working context is rewritten to $\mathbf{M}=[m_0,u,\mathsf{IndexedSummary}]$. The $t_i$ and $o_i$ are the thinking with tool call request and the tool output at step $i$, respectively.

工作上下文压缩示例。 我们首先说明索引经验记忆的压缩操作如何将不断增长的上下文重写为紧凑、以指针为主的状态。 这里展示一个具体的压缩前后示例：多轮推理和工具输出以稳定索引归档到 $\mathcal{D}$ 中，而工作上下文被重写为 $\mathbf{M}=[m_0,u,\mathsf{IndexedSummary}]$。 $t_i$ 和 $o_i$ 分别表示第 $i$ 步中包含工具调用请求的思考，以及工具输出。

Example for Compression with Indexed Experience Memory (Before/After)
索引经验记忆的压缩示例（压缩前后）

Context Before compression: / 压缩前的上下文：

$$\begin{array}{rl}& \mathbf{M}=[m_0,u,t_1,o_1,t_2,o_2,t_3,o_3,\\ & t_4,o_4,t_5,o_5,t_6,o_6]\end{array}$$

• $m_0$ System prompt: tool-usage rules and context management constraints.
• $u$ User task: “SI._collect_factor_and_dimension() cannot properly detect that exponent is dimensionless” with a repro using exp and a failing ValueError......
• $t_1$ Reasoning: orient in repo to find units code; Tool call: list /testbed tree (2 levels). (approx. 350 tokens)
• $o_1$ Repo snapshot: /testbed structure with key paths under sympy/physics/units. (approx. 600 tokens)
• $t_2$ Reasoning: locate the core method; Tool call: search _collect_factor_and_dimension in sympy/physics/units. (approx. 280 tokens)
• $o_2$ Search hits: 28 matches, including sympy/physics/units/unitsystem.py (9 matches). (approx. 240 tokens)
• $t_3$ Reasoning: jump to exact definition; Tool call: search for def _collect_factor_and_dimension in unitsystem.py. (approx. 220 tokens)
• $o_3$ Exact location: match at line 152 in unitsystem.py. (approx. 120 tokens)
• $t_4$ Reasoning: inspect dimension handling in context; Tool call: view unitsystem.py lines 100–160. (approx. 420 tokens)
• $o_4$ Code excerpt: get_dimensional_expr and start of _collect_factor_and_dimension. (approx. 900 tokens)
• $t_5$ Reasoning: inspect full Function branch; Tool call: view unitsystem.py lines 140–206. (approx. 420 tokens)
• $o_5$ Code excerpt: full _collect_factor_and_dimension showing Function branch return inconsistency. (approx. 1100 tokens)
• $t_6$ Reasoning: preserve reproducible failing case; Tool call: create reproduce_issue.py. (approx. 280 tokens)
• $o_6$ Reproducer content mirroring the GitHub issue (exp on dimensionless expr). (approx. 450 tokens)

• $m_0$ 系统提示：工具使用规则和上下文管理约束。
• $u$ 用户任务：“SI._collect_factor_and_dimension() 无法正确检测出指数是无量纲的”，并给出了使用 exp 且触发 ValueError 的复现案例……
• $t_1$ 推理：在仓库中定位单位相关代码；工具调用：列出 /testbed 目录树（2 层）。（约 350 个 token）
• $o_1$ 仓库快照：/testbed 的结构，以及 sympy/physics/units 下的关键路径。（约 600 个 token）
• $t_2$ 推理：定位核心方法；工具调用：在 sympy/physics/units 中搜索 _collect_factor_and_dimension。（约 280 个 token）
• $o_2$ 搜索结果：28 处匹配，包括 sympy/physics/units/unitsystem.py 中的 9 处匹配。（约 240 个 token）
• $t_3$ 推理：跳转到确切定义；工具调用：在 unitsystem.py 中搜索 def _collect_factor_and_dimension。（约 220 个 token）
• $o_3$ 精确位置：匹配位于 unitsystem.py 第 152 行。（约 120 个 token）
• $t_4$ 推理：结合上下文检查量纲处理；工具调用：查看 unitsystem.py 第 100–160 行。（约 420 个 token）
• $o_4$ 代码片段：get_dimensional_expr 和 _collect_factor_and_dimension 的开头部分。（约 900 个 token）
• $t_5$ 推理：检查完整的 Function 分支；工具调用：查看 unitsystem.py 第 140–206 行。（约 420 个 token）
• $o_5$ 代码片段：完整的 _collect_factor_and_dimension，展示了 Function 分支返回值的不一致。（约 1100 个 token）
• $t_6$ 推理：保存可复现的失败案例；工具调用：创建 reproduce_issue.py。（约 280 个 token）
• $o_6$ 复现脚本内容与 GitHub issue 对应，即对无量纲表达式使用 exp。（约 450 个 token）

Context After compression: / 压缩后的上下文：

$$\mathbf{M}=[m_0,u,\mathsf{IndexedSummary}]$$

IndexedSummary (approx. 300 tokens):

CurrentStatus: Next, inspect sympy/physics/units/dimensions.py for Dimension and DimensionSystem to ensure proper construction of Dimension(1). Then patch unitsystem.py Function branch to return (expr.func of factors, Dimension(1)) when args are dimensionless; otherwise, decide correct behavior (likely raise error or compute dimension expression). Finally, run reproduce_issue.py before and after patch to confirm fix.

IndexMap:

• ctx_repo_snapshot_001 /testbed directory snapshot (2-level listing) for orientation; use to avoid re-running repo view.
• ctx_units_search_001 search hits for "_collect_factor_and_dimension" across units module; shows key files and counts.
• ctx_units_method_loc_001 precise location of def _collect_factor_and_dimension in unitsystem.py (line 152) to jump directly.
• ctx_units_code_excerpt_001 excerpt (lines 100-160) of unitsystem.py showing get_dimensional_expr and beginning of _collect_factor_and_dimension; crucial for understanding dimension handling and function branch logic.
• ctx_units_code_excerpt_002 excerpt (lines 140-206) of unitsystem.py covering the full _collect_factor_and_dimension logic including Function branch (lines 192-196); identifies inconsistency in return type for Function.
• ctx_repro_script_001 reproduce_issue.py content mirroring the GitHub issue to validate bug and later confirm fix.

IndexedSummary（约 300 个 token）：

CurrentStatus（当前状态）： 接下来，检查 sympy/physics/units/dimensions.py 中的 Dimension 和 DimensionSystem，以确保正确构造 Dimension(1)。然后修补 unitsystem.py 的 Function 分支：当参数无量纲时，返回 (expr.func of factors, Dimension(1))；否则确定正确行为，可能是抛出错误或计算量纲表达式。最后，在修补前后运行 reproduce_issue.py，确认修复有效。

IndexMap（索引映射）：

• ctx_repo_snapshot_001 /testbed 目录快照（2 层目录列表），用于了解仓库结构，避免重新查看仓库。
• ctx_units_search_001 在 units 模块中搜索“_collect_factor_and_dimension”的结果，展示关键文件及匹配次数。
• ctx_units_method_loc_001 unitsystem.py 中 def _collect_factor_and_dimension 的精确位置（第 152 行），便于直接跳转。
• ctx_units_code_excerpt_001 unitsystem.py 的代码片段（第 100-160 行），展示 get_dimensional_expr 和 _collect_factor_and_dimension 的开头；对理解量纲处理及函数分支逻辑至关重要。
• ctx_units_code_excerpt_002 unitsystem.py 的代码片段（第 140-206 行），涵盖完整的 _collect_factor_and_dimension 逻辑，包括 Function 分支（第 192-196 行）；指出了 Function 返回类型的不一致。
• ctx_repro_script_001 reproduce_issue.py 的内容，与 GitHub issue 对应，用于验证缺陷并在之后确认修复。

Algorithm 1: Memex Agent Loop

1. Input: 任务指令 u、最大步数 T_max、上下文阈值 τ
2. Output: 提交的答案 answer

Tool set: T = {CompressExperience(·), ReadExperience(·), Finish(·), OtherTool(·)}

3. function Execute(u, T_max, τ)
4. M ← [m₀, u] ▷ 系统提示与任务指令，永不压缩
5. D ← ∅ ▷ 记忆池
6. answer ← ∅
7. for t = 1 to T_max do
8. M ← M ⊕ [ContextStatus(M, τ)]
9. (z_t, c_t) ← π_agent(M) ▷ z_t：思考；c_t：工具调用（含 Finish/Compress/Read）
10. M ← M ⊕ [z_t, c_t]
11. if c_t = CompressExperience(IndexedSummary, MemoryBlocks) then
12. for (index, content) in MemoryBlocks do
13. D[index] ← content
14. M ← [m₀, u, IndexedSummary]
15. else if c_t = ReadExperience(index) then
16. o_t ← D[index]
17. M ← M ⊕ [o_t] ▷ 将检索内容作为新消息
18. else if c_t = Finish(y) then
19. answer ← y
20. return answer
21. else
22. o_t ← ExecuteTool(c_t)
23. M ← M ⊕ [o_t] ▷ 将环境或工具输出作为新消息
24. return MaxStepReached

Agent Loop with Indexed Experience Memory. Algorithm 1 specifies the full execution loop. Initialization sets $\mathbf{M}\leftarrow [m_0,u]$, $\mathcal{D}\leftarrow \mathrm{\varnothing }$, and $answer\leftarrow \varnothing$ (lines 4–6), where the two anchors $m_0$ (system prompt) and $u$ (task instruction) are never compressed. At each step $t$ (line 7), the system first appends ContextStatus$(\mathbf{M},\tau )$ (line 8), a deterministic message computed from the current context window $\mathbf{M}$ that reports its working context token usages and the threshold $\tau$ (e.g., “[Context Status: working context tokens=6932, threshold=8000]”). The agent then emits thinking $z_t$ and a tool call $c_t$ conditioned on $\mathbf{M}$ (line 9), and appends them to the working context (line 10).

具有索引经验记忆的智能体循环。 算法1 给出了完整的执行循环。 初始化时设置 $\mathbf{M}\leftarrow [m_0,u]$、$\mathcal{D}\leftarrow \mathrm{\varnothing }$ 和 $answer\leftarrow \varnothing$（第 4–6 行），其中两个锚点 $m_0$（系统提示）和 $u$（任务指令）永远不会被压缩。 在每一步 $t$（第 7 行），系统首先追加 ContextStatus$(\mathbf{M},\tau )$（第 8 行），这是一条根据当前上下文窗口 $\mathbf{M}$ 计算出的确定性消息，报告工作上下文的 token 用量和阈值 $\tau$，例如“[Context Status: working context tokens=6932, threshold=8000]”。 随后，智能体以 $\mathbf{M}$ 为条件生成思考 $z_t$ 和工具调用 $c_t$（第 9 行），并将它们追加到工作上下文（第 10 行）。

If $c_t=$ CompressExperience(IndexedSummary, MemoryBlocks), the agent archives each $(\mathsf{index},\mathsf{content})\in \mathsf{MemoryBlocks}$ into the external store via $\mathcal{D}[\mathsf{index}]\leftarrow \mathsf{content}$ (lines 12–13), and rewrites the working context to $\mathbf{M}\leftarrow [m_0,u,\mathsf{IndexedSummary}]$ (line 14). If $c_t=\mathrm{ReadExperience}(\mathsf{index})$, the agent dereferences the store via $o_t\leftarrow \mathcal{D}[\mathsf{index}]$ (line 16) and appends the retrieved block as a new message $\mathbf{M}\leftarrow \mathbf{M}\oplus [o_t]$ (line 17). If $c_t=\mathrm{Finish}(y)$, the agent sets $answer\leftarrow y$ (line 19) and returns (line 20). Otherwise, it executes other tools (line 22) and appends the observation to $\mathbf{M}$ (line 23). If no Finish occurs within $T_{max}$ steps, the procedure terminates (line 24).

若 $c_t=$ CompressExperience(IndexedSummary, MemoryBlocks)，智能体通过 $\mathcal{D}[\mathsf{index}]\leftarrow \mathsf{content}$，将每个 $(\mathsf{index},\mathsf{content})\in \mathsf{MemoryBlocks}$ 归档到外部存储（第 12–13 行），并将工作上下文重写为 $\mathbf{M}\leftarrow [m_0,u,\mathsf{IndexedSummary}]$（第 14 行）。 若 $c_t=\mathrm{ReadExperience}(\mathsf{index})$，智能体通过 $o_t\leftarrow \mathcal{D}[\mathsf{index}]$ 解引用存储（第 16 行），并将检索到的块作为新消息追加，即 $\mathbf{M}\leftarrow \mathbf{M}\oplus [o_t]$（第 17 行）。 若 $c_t=\mathrm{Finish}(y)$，智能体设置 $answer\leftarrow y$（第 19 行）并返回（第 20 行）。 否则，它执行其他工具（第 22 行），并将观测追加到 $\mathbf{M}$（第 23 行）。 若在 $T_{max}$ 步内未发生 Finish，该过程终止（第 24 行）。

3.3. MemexRL: Learning Indexed Experience Memory Enabled Agent with RL

The agent loop in Algorithm 1 treats memory operations (CompressExperience, ReadExperience) as tool calls in the same action space as environment tools and Finish. A compression step must decide what to archive, how to index it, and what to keep in-context: archiving the right evidence under stable indices and writing an actionable index-map summary may only pay off many steps later by enabling precise dereferencing, preventing redundant tool calls, and avoiding context overflow. Conversely, missing or ambiguous indices can silently derail future reasoning even when the summary appears locally plausible. These decisions are inherently difficult to enforce with prompt rules alone, since the utility of an index entry is revealed only when a downstream step needs to retrieve exactly that artifact. We therefore learn memory behaviors jointly with task-solving behaviors using reinforcement learning. We optimize the agent policy with MemexRL, a GRPO-style update specialized to tool-using agents with Indexed Experience Memory, encouraging trajectories that solve the task while using compression and dereferencing efficiently and correctly.

算法1 中的智能体循环将记忆操作（CompressExperience、ReadExperience）视为工具调用，与环境工具和 Finish 位于同一动作空间。 一次压缩必须决定 归档什么、如何建立索引，以及 在上下文中保留什么：将正确的证据以稳定索引归档，并写出可据以行动的索引映射摘要，可能要到许多步骤之后，才会通过精确解引用、防止冗余工具调用和避免上下文溢出产生回报。 反过来，即使摘要局部看似合理，缺失或模糊的索引仍可能在不易察觉的情况下使后续推理偏离正轨。 仅靠提示规则本就难以强制做好这些决策，因为只有当下游步骤恰好需要检索某个产物时，相应索引条目的效用才会显现。 因此，我们使用强化学习，将记忆行为与任务求解行为联合学习。 我们使用 MemexRL 优化智能体策略，这是一种专为具有索引经验记忆的工具使用智能体设计的 GRPO 式更新方法，鼓励产生既能解决任务，又能高效、正确地使用压缩与解引用的轨迹。

Reward design for MemexRL. We define an episode-level return that combines task success with three memory-efficiency penalties, each normalized to $[0,1]$:

MemexRL 的奖励设计。 我们定义回合级回报，将任务成功与三项记忆效率惩罚相结合，每项惩罚均归一化到 $[0,1]$：

$$\begin{array}{}\text{(1)}& \begin{array}{rl}& R=R_{\text{task}}-P_{\text{context}}\\ & -P_{\text{redundancy}}-P_{\text{format}}.\end{array}\end{array}$$

Context overflow penalty. Let $C_t$ denote the working context size (in tokens) at step $t$, excluding steps where compression was triggered. The penalty accumulates overflow tokens beyond threshold $\tau$ across all $T$ steps: $P_{\text{context}}=min(1,\frac{\sum _{t=1}^{T}max(0,C_t-\tau )}{\tau \cdot T})$. Normalizing by the maximum reasonable overflow $\tau \cdot T$ bounds the penalty in $[0,1]$. This encourages the agent to proactively invoke CompressExperience before context grows excessively, rather than waiting until forced truncation degrades performance.

上下文溢出惩罚。 令 $C_t$ 表示第 $t$ 步的工作上下文大小，以 token 为单位，不计触发压缩的步骤。 该惩罚累计全部 $T$ 步中超出阈值 $\tau$ 的 token 数：$P_{\text{context}}=min(1,\frac{\sum _{t=1}^{T}max(0,C_t-\tau )}{\tau \cdot T})$。 通过最大合理溢出量 $\tau \cdot T$ 进行归一化，使惩罚限制在 $[0,1]$ 内。 这鼓励智能体在上下文过度增长之前主动调用 CompressExperience，而不是等到强制截断导致性能下降。

Redundant tool call penalty. Let $N_{\text{redundant}}$ count tool calls with identical (tool name, arguments) signatures that were previously executed, when no state-modifying operation has occurred since the last identical call. The penalty is: $P_{\text{redundancy}}=\frac{N_{\text{redundant}}}{N_{\text{tool\_call}}}$, where $N_{\text{tool\_call}}$ is the total number of non-memory tool calls in the trajectory. This penalty discourages repetitive exploration patterns—such as viewing the same file region multiple times without edits—and encourages the agent to utilize ReadExperience to recall previously observed information instead of re-executing identical queries.

冗余工具调用惩罚。 令 $N_{\text{redundant}}$ 统计具有相同（工具名称、参数）签名、且此前已执行过的工具调用次数，前提是自上一次相同调用以来没有发生修改状态的操作。 惩罚为 $P_{\text{redundancy}}=\frac{N_{\text{redundant}}}{N_{\text{tool\_call}}}$，其中 $N_{\text{tool\_call}}$ 是轨迹中非记忆工具调用的总次数。 这一惩罚抑制重复探索模式，例如在没有编辑的情况下多次查看同一文件区域，并鼓励智能体利用 ReadExperience 回忆先前观察到的信息，而不是重新执行相同查询。

Format error penalty. Let $N_{\text{malformed}}$ count malformed tool calls detected in the agent's raw output, including: (1) tool call tag mismatches (e.g., <tool_call> without closing </tool_call>), (2) invalid JSON within tool call tags, and (3) missing required fields such as name or arguments. The penalty is: $P_{\text{format}}=\frac{N_{\text{malformed}}}{N_{\text{tool\_call}}}$, where $N_{\text{tool\_call}}$ is the number of steps where the agent attempted to invoke a tool. This provides a direct learning signal for generating syntactically correct tool invocations, which is particularly important for base models without prior tool-use fine-tuning.

格式错误惩罚。 令 $N_{\text{malformed}}$ 统计在智能体原始输出中检测到的格式错误工具调用次数，包括：(1) 工具调用标签不匹配，例如有 <tool_call> 却没有闭合的 </tool_call>；(2) 工具调用标签内存在无效 JSON；(3) 缺少 name 或 arguments 等必填字段。 惩罚为 $P_{\text{format}}=\frac{N_{\text{malformed}}}{N_{\text{tool\_call}}}$，其中 $N_{\text{tool\_call}}$ 是智能体尝试调用工具的步骤数。 这为生成语法正确的工具调用提供了直接学习信号，对于此前未经过工具使用微调的基础模型尤其重要。

Segmented Trajectory Processing. When compression occurs during a trajectory, we segment the trajectory at compression boundaries and process each segment as an independent training sample while preserving credit assignment to the final outcome. This design is particularly natural for autoregressive models, whose token probabilities are always conditioned on the current prefix. Once compression is triggered, the prefix for all subsequent generation changes: instead of continuing from the full pre-compression history, the model now conditions on a compressed summary together with the new interactions that follow. In this sense, compression does not merely shorten the context; it changes the conditioning prefix that defines the downstream policy.

分段轨迹处理。 当轨迹中发生压缩时，我们在压缩边界处对轨迹分段，将每个片段作为独立训练样本处理，同时保留与最终结果之间的信用分配关系。 这一设计对于自回归模型尤其自然，因为其 token 概率始终以当前前缀为条件。 一旦触发压缩，所有后续生成的前缀都会改变：模型不再从压缩前的完整历史继续生成，而是以压缩摘要及其后续的新交互为条件。 从这个意义上说，压缩不仅仅是缩短上下文；它改变了定义下游策略的条件前缀。

Concretely, suppose an agent compresses $k$ times during a trajectory, producing $k+1$ segments $\{S_0,S_1,\dots ,S_k\}$. Each segment $S_i$ maintains its own context. $S_0$ contains the full pre-compression history, while each subsequent segment $S_i$ for $i>0$ contains the compressed summary from the previous segment together with new interactions, i.e.,

具体而言，假设智能体在一条轨迹中压缩了 $k$ 次，产生 $k+1$ 个片段 $\{S_0,S_1,\dots ,S_k\}$。 每个片段 $S_i$ 维护自己的上下文。 $S_0$ 包含压缩前的完整历史，而对于 $i>0$，每个后续片段 $S_i$ 包含前一片段的压缩摘要和新的交互，即

$$\begin{array}{rl}& S_i=[\mathtt{system},\mathtt{task},{\mathtt{summary}}_{i-1},\\ & z_{i1},c_{i1},o_{i1},z_{i2},c_{i2},o_{i2},\dots ].\end{array}$$

Here, $z_{ij}$ denotes the model's intermediate reasoning or textual generation at the $j$-th interaction step of segment $S_i$, $c_{ij}$ denotes the corresponding tool call or action command, and $o_{ij}$ denotes the resulting observation returned by the environment or tool. Thus, each segment consists of a sequence of reasoning–action–observation tuples appended after the inherited summary context. During training, we flatten all segments across the batch, allowing each segment to be tokenized and optimized independently under its own context window. Crucially, all segments originating from the same trajectory share the identical terminal reward $R$, which preserves credit assignment to earlier compression decisions through group-relative advantage estimation in GRPO. Thus, the model can learn whether a compression-induced prefix helped or harmed the eventual outcome, and the write policy receives gradient signal from the shared terminal reward even though its effect may only appear in later segments.

这里，$z_{ij}$ 表示模型在片段 $S_i$ 的第 $j$ 个交互步骤中的中间推理或文本生成，$c_{ij}$ 表示相应的工具调用或动作命令，$o_{ij}$ 表示环境或工具返回的观测结果。 因此，每个片段都由继承的摘要上下文，以及在其后追加的一系列推理–动作–观测三元组组成。 训练时，我们将批次中的所有片段展平，使每个片段都能在各自的上下文窗口内独立分词和优化。 关键在于，来自同一轨迹的所有片段共享完全相同的终局奖励 $R$，从而通过 GRPO 中的组相对优势估计，保留对较早压缩决策的信用分配。 因此，模型能够学习由压缩产生的前缀究竟有助于还是损害了最终结果，而写入策略能够从共享终局奖励中获得梯度信号，即使其影响只在后续片段中才显现。

Automatic Compression Triggering. Unlike prior work that enforces compression automatically when context length exceeds a threshold, we adopt a soft triggering approach that automatically monitors context status and prompts the agent to compress voluntarily. Specifically, at each step, we append a context status indicator to the observation: [Context Status: working=w, total=t, threshold=L]. When the working context $w$ approaches or exceeds $L$, additional warnings are automatically injected (e.g., “working > threshold”). This design transforms context management from a system-enforced constraint into a learnable skill.

自动压缩触发。 以往工作在上下文长度超出阈值时自动强制压缩；与之不同，我们采用软触发方式，自动监测上下文状态并提示智能体主动压缩。 具体而言，每一步中，我们都会在观测后追加一个上下文状态指示器：[Context Status: working=w, total=t, threshold=L]。 当工作上下文 $w$ 接近或超过 $L$ 时，系统会自动注入额外警告，例如 “working > threshold”。 这一设计将上下文管理从系统强制执行的约束转化为可学习的技能。

The agent must learn to (1) recognize when compression is beneficial, (2) decide the optimal timing for compression based on task semantics rather than arbitrary token counts, and (3) generate high-quality summaries that preserve task-relevant information. Through reinforcement learning with context overflow penalties, the agent is incentivized to compress proactively while maintaining task performance. This approach offers greater flexibility than hard thresholds—for instance, the agent may defer compression if it anticipates task completion within a few steps, or compress earlier at natural semantic boundaries.

智能体必须学会：(1) 识别何时压缩有益；(2) 根据任务语义而非任意的 token 数量，决定最佳压缩时机；(3) 生成保留任务相关信息的高质量摘要。 通过带有上下文溢出惩罚的强化学习，智能体受到激励，在维持任务性能的同时主动压缩。 这种方式比硬阈值更灵活，例如，如果智能体预计再过几步即可完成任务，它可以推迟压缩；也可以在自然的语义边界处更早压缩。

Overall learning algorithm. For each training instance, we sample a group of $G$ rollouts $\{y^{(g)}{\}}_{g=1}^G$ from the current policy, where each rollout is a sequence of model tokens that includes both the agent’s thinking text and a tool call at each step (including memory tools and Finish). We execute the resulting tool calls to obtain a scalar return $R^{(g)}$ using Equation 1, and compute a group-relative, normalized advantage

整体学习算法。 对每个训练实例，我们从当前策略采样一组包含 $G$ 条轨迹的 $\{y^{(g)}{\}}_{g=1}^G$，其中每条轨迹都是模型 token 序列，在每一步中包含智能体的思考文本和一次工具调用，包括记忆工具与 Finish。 我们执行生成的工具调用，使用 公式1 获得标量回报 $R^{(g)}$，并计算经归一化的组相对优势：

$$\begin{array}{}\text{(2)}& A^{(g)}=\frac{R^{(g)}-\mathrm{mean}(\{R^{(h)}{\}}_{h=1}^G)}{\mathrm{std}(\{R^{(h)}{\}}_{h=1}^G)+ϵ}.\end{array}$$

Let ${\pi }_{\theta }$ be the current policy and ${\pi }_{{\theta }_{\text{old}}}$ be the sampling policy for the group. For each rollout, we form the token-level ratio $r_t^{(g)}={\pi }_{\theta }(a_t^{(g)}\mid s_t^{(g)})/{\pi }_{{\theta }_{\text{old}}}(a_t^{(g)}\mid s_t^{(g)})$ and apply a PPO-style clipped surrogate objective, optionally regularized by a KL penalty to a reference policy ${\pi }_{\text{ref}}$:

令 ${\pi }_{\theta }$ 为当前策略，${\pi }_{{\theta }_{\text{old}}}$ 为该组的采样策略。 对每条轨迹，我们构造 token 级比率 $r_t^{(g)}={\pi }_{\theta }(a_t^{(g)}\mid s_t^{(g)})/{\pi }_{{\theta }_{\text{old}}}(a_t^{(g)}\mid s_t^{(g)})$，并应用 PPO 式裁剪替代目标，还可选择通过相对于参考策略 ${\pi }_{\text{ref}}$ 的 KL 惩罚进行正则化：

$$\begin{array}{}\text{(3)}& \begin{array}{rl}& \underset{\theta }{max}\frac{1}{G}\sum _{g=1}^G\sum _t\\ & min(r_t^{(g)}{A}^{(g)},\\ & \mathrm{clip}(r_t^{(g)},1-\eta ,1+\eta )A^{(g)})\\ & -\beta \mathrm{KL}({\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}}).\end{array}\end{array}$$

Because $R^{(g)}$ explicitly scores context-budget violations, redundant tool use, and oversized archival blocks, the group-relative advantage in Equation 2 encourages the policy to discover memory behaviors that improve long-horizon returns: compressing into pointer-heavy index maps that remain usable later, archiving high-value artifacts under stable indices, and selectively retrieving evidence instead of re-running tools. In practice, we warm-start MemexRL from supervised demonstrations of well-formed $\sigma$ and block sets $\mathsf{Blocks}$, then refine with RL under the same message-only interface.

由于 $R^{(g)}$ 显式对超出上下文预算、冗余工具使用和过大的归档块进行评分，公式2 中的组相对优势会鼓励策略发现能够提高长时程回报的记忆行为：将上下文压缩为之后仍可使用、以指针为主的索引映射，以稳定索引归档高价值产物，以及有选择地检索证据而非重新运行工具。 实践中，我们先使用格式良好的 $\sigma$ 和块集合 $\mathsf{Blocks}$ 的监督示范对 MemexRL 进行热启动，再在同一个仅通过消息交互的接口下使用 RL 进一步优化。

4. Theoretical Analysis of Why the Memex Loop Scales to Long-Horizon Agents

This section examines the theoretical potential of the Memex agent loop. In particular, we ask whether Indexed Experience Memory can, in principle, support two desirable properties at the same time: preserving optimal decision quality without conditioning on the full message history, and keeping the agent's effective in-context computation bounded as the message history grows. Our goal is not to prove that MemexRL always learns such summaries in practice, but to characterize the regime in which indexed summaries and bounded dereferencing are sufficient to support accurate decisions with bounded working context.

本节考察 Memex 智能体循环的理论潜力。 具体而言，我们探讨索引经验记忆在原理上能否同时支持两个理想性质：无需以完整消息历史为条件也能保持最优决策质量，以及在消息历史增长时，让智能体有效的上下文内计算保持有界。 我们的目标不是证明 MemexRL 在实践中总能学会这样的摘要，而是刻画这样一种情形：索引摘要与有界解引用足以在有界工作上下文下支持准确决策。

Property 1: Preserving decision quality with bounded dereferencing. Let $M_t^{\mathrm{full}}$ denote the full message history up to step $t$, and let $({\sigma }_t,D_t)$ be the Memex state induced by $M_t^{\mathrm{full}}$, where ${\sigma }_t$ is the in-context indexed summary and $D_t:\mathsf{index}\mapsto \mathsf{content}$ is the external experience store.

性质1：通过有界解引用保持决策质量。 令 $M_t^{\mathrm{full}}$ 表示截至第 $t$ 步的完整消息历史，令 $({\sigma }_t,D_t)$ 为由 $M_t^{\mathrm{full}}$ 产生的 Memex 状态，其中 ${\sigma }_t$ 是上下文内索引摘要，$D_t:\mathsf{index}\mapsto \mathsf{content}$ 是外部经验存储。

We say that the indexed summary is sufficient for decision making if it does not need to contain all past evidence itself, but can instead point to the small subset of archived evidence that matters for the current decision.

如果索引摘要本身无需包含所有过去的证据，而是能够指向与当前决策有关的一小部分归档证据，我们就称它对决策而言是充分的。

Definition 3 (Decision-sufficient indexed summary). We say that ${\sigma }_t$ is $B$-bounded decision-sufficient if there exist an index selector $g$ and a decision function $\mu$ such that $|g({\sigma }_t)|\le B$ and ${\pi }^{\ast }(\cdot \mid M_t^{\mathrm{full}})=\mu (\cdot \mid {\sigma }_t,\{D_t[i]{\}}_{i\in g({\sigma }_t)})$, where ${\pi }^{\ast }$ is an optimal policy that conditions on the full message history $M_t^{\mathrm{full}}$.

定义3（决策充分的索引摘要）。 若存在索引选择器 $g$ 和决策函数 $\mu$，使得 $|g({\sigma }_t)|\le B$ 且 ${\pi }^{\ast }(\cdot \mid M_t^{\mathrm{full}})=\mu (\cdot \mid {\sigma }_t,\{D_t[i]{\}}_{i\in g({\sigma }_t)})$，则称 ${\sigma }_t$ 是 $B$ 有界决策充分的；其中，${\pi }^{\ast }$ 是以完整消息历史 $M_t^{\mathrm{full}}$ 为条件的最优策略。

This definition formalizes the desired Memex property. Although the full message history may be arbitrarily long, the optimal action can still be recovered from a compact indexed summary together with at most $B$ explicitly dereferenced blocks from the archive.

这一定义形式化了我们期望 Memex 具备的性质。 尽管完整消息历史可以任意长，仍然可以根据紧凑的索引摘要，加上从档案中显式解引用的至多 $B$ 个块，恢复最优动作。

Proposition 1 (Memex can match a full-context optimal policy). Let $J(\pi )$ denote the expected return of policy $\pi$. Assume that ${\sigma }_t$ is $B$-bounded decision-sufficient for every step $t$. Then there exists a Memex policy ${\pi }_{\mathrm{IEM}}$ that conditions only on ${\sigma }_t$ and uses at most $B$ calls to ReadExperience$(\cdot )$ per step such that $J({\pi }_{\mathrm{IEM}})=J({\pi }^{\ast })$.

命题1（Memex 可以匹配基于完整上下文的最优策略）。 令 $J(\pi )$ 表示策略 $\pi$ 的期望回报。 假设在每一步 $t$，${\sigma }_t$ 都是 $B$ 有界决策充分的。 那么，存在一个仅以 ${\sigma }_t$ 为条件、每步至多调用 $B$ 次 ReadExperience$(\cdot )$ 的 Memex 策略 ${\pi }_{\mathrm{IEM}}$，满足 $J({\pi }_{\mathrm{IEM}})=J({\pi }^{\ast })$。

Interpretation. This result says that if the indexed summary plus a bounded number of dereferences capture all decision-relevant evidence, then Memex loses no decision quality relative to an agent that always conditions on the full message history.

解释。 这一结果表明，如果索引摘要加上有界次数的解引用能够捕捉全部与决策有关的证据，那么相对于始终以完整消息历史为条件的智能体，Memex 不会损失决策质量。

Proof sketch. At each step, the policy first uses $g({\sigma }_t)$ to select at most $B$ indices, dereferences the corresponding blocks from $D_t$, and then applies $\mu$ to $({\sigma }_t,\{D_t[i]{\}}_{i\in g({\sigma }_t)})$. By definition, this reproduces the same action distribution as the optimal full-context policy ${\pi }^{\ast }$ at every step, and therefore achieves the same expected return. Detailed proof can be found in Appendix A.

证明概要。 在每一步，策略首先使用 $g({\sigma }_t)$ 选择至多 $B$ 个索引，从 $D_t$ 中解引用相应的块，然后将 $\mu$ 应用于 $({\sigma }_t,\{D_t[i]{\}}_{i\in g({\sigma }_t)})$。 根据定义，这会在每一步复现与最优完整上下文策略 ${\pi }^{\ast }$ 相同的动作分布，因此获得相同的期望回报。 详细证明见 附录 A。

Property 2: Keeping working context bounded as trajectory grows. The second property concerns efficiency. Memex is useful only if it keeps the working context small even when the full history becomes very long. Let $C_t^{\mathrm{full}}\triangleq |M_t^{\mathrm{full}}|$ denote the token length of the full message history. Under Memex, the model does not need to keep all of $M_t^{\mathrm{full}}$ in its working context. Instead, the working context at step $t$ consists only of the indexed summary ${\sigma }_t$ together with the dereferenced blocks used at that step. We therefore define the working-context length as $C_t^{\mathrm{work}}\triangleq |{\sigma }_t|+\sum _{i\in I_t}|D_t[i]|$, where $I_t$ is the set of indices read at step $t$.

性质2：轨迹增长时工作上下文保持有界。 第二个性质涉及效率。 只有在完整历史变得很长时仍能保持较小的工作上下文，Memex 才有用。 令 $C_t^{\mathrm{full}}\triangleq |M_t^{\mathrm{full}}|$ 表示完整消息历史的 token 长度。 在 Memex 下，模型无需将全部 $M_t^{\mathrm{full}}$ 保留在工作上下文中。 相反，第 $t$ 步的工作上下文只由索引摘要 ${\sigma }_t$ 和该步所使用的解引用块组成。 因此，我们将工作上下文长度定义为 $C_t^{\mathrm{work}}\triangleq |{\sigma }_t|+\sum _{i\in I_t}|D_t[i]|$，其中 $I_t$ 是第 $t$ 步读取的索引集合。

Proposition 2 (Memex keeps working context bounded). Assume that for all $t$, the summary length satisfies $|{\sigma }_t|\le {\tau }_{\sigma }$, the number of dereferenced indices satisfies $|I_t|\le B$, and each retrieved block satisfies $|D_t[i]|\le L$. Then $C_t^{\mathrm{work}}\le {\tau }_{\sigma }+BL\triangleq C_{\mathrm{work}}^{max}$. Consequently, the compression ratio ${\rho }_t\triangleq \frac{C_t^{\mathrm{full}}}{C_t^{\mathrm{work}}}$ satisfies ${\rho }_t\ge \frac{C_t^{\mathrm{full}}}{C_{\mathrm{work}}^{max}}$. In particular, as the full message history length $C_t^{\mathrm{full}}$ grows, the working context remains bounded while the compression ratio grows without bound.

命题2（Memex 使工作上下文保持有界）。 假设对于所有 $t$，摘要长度满足 $|{\sigma }_t|\le {\tau }_{\sigma }$，解引用的索引数量满足 $|I_t|\le B$，且每个检索块满足 $|D_t[i]|\le L$。 则有 $C_t^{\mathrm{work}}\le {\tau }_{\sigma }+BL\triangleq C_{\mathrm{work}}^{max}$。 因此，压缩比 ${\rho }_t\triangleq \frac{C_t^{\mathrm{full}}}{C_t^{\mathrm{work}}}$ 满足 ${\rho }_t\ge \frac{C_t^{\mathrm{full}}}{C_{\mathrm{work}}^{max}}$。 特别地，随着完整消息历史长度 $C_t^{\mathrm{full}}$ 增长，工作上下文仍保持有界，而压缩比无界增长。

Interpretation. This result formalizes the central efficiency advantage of Memex. Even as the full message history keeps growing, the agent only needs to maintain a bounded working context, provided that the summary remains compact and only a bounded number of archived blocks are explicitly dereferenced at each step. Since the system prompt $m_0$ and task instruction $u$ are fixed, bounded working context also implies bounded effective in-context computation. Detailed proof can be found in Appendix A.

解释。 这一结果形式化了 Memex 的核心效率优势。 即使完整消息历史持续增长，只要摘要保持紧凑，且每一步只显式解引用有界数量的归档块，智能体就只需维护有界的工作上下文。 由于系统提示 $m_0$ 和任务指令 $u$ 固定，有界工作上下文也意味着有效的上下文内计算有界。 详细证明见 附录 A。

5. Empirical Results for MemexRL

The previous section establishes the theoretical potential of the Memex loop: in principle, indexed experience memory can preserve decision quality while keeping the working context bounded. We now turn to empirical evaluation to examine whether this potential can be realized in practice by our training method MemexRL.

上一节确立了 Memex 循环的理论潜力：在原理上，索引经验记忆能够在保持工作上下文有界的同时保留决策质量。 现在，我们转向实证评估，考察训练方法 MemexRL 能否在实践中实现这一潜力。

5.1. Experiments Setup

LLM Models. We use Qwen3-30B-A3B-Thinking-2507 in our experiments. It is a Mixture-of-Experts (MoE) model with about 30B total parameters, 128 experts per layer with top-8 routing, and approximately 3B active parameters per token. It has strong tool-use understanding and instruction-following capabilities, which allow it to understand the memory tools provided in the prompt and produce useful CompressExperience and ReadExperience usage patterns before RL training.

LLM 模型。 实验使用 Qwen3-30B-A3B-Thinking-2507。 这是一个混合专家（MoE）模型，总参数量约为 300 亿，每层有 128 个专家，采用 top-8 路由，每个 token 的活跃参数量约为 30 亿。 它具有较强的工具使用理解能力和指令遵循能力，因此在 RL 训练之前就能理解提示中提供的记忆工具，并产生有用的 CompressExperience 与 ReadExperience 使用模式。

图2：训练过程中采样轨迹的任务成功率。智能体的任务成功率从约 20% 提升到 90% 以上，表明 MemexRL 训练有效地教会模型使用 Memex 智能体循环解决任务。

图3：训练过程中采样轨迹的总惩罚。惩罚幅度从 −0.4 减小至约 −0.1，表明智能体学会了更好地执行任务，并利用 CompressExperience 和 ReadExperience 有策略地压缩峰值工作上下文长度。

(a) 任务成功率

(b) 峰值工作上下文长度

图4：MemexRL 的有效性。(a) 任务成功率从 24.2% 提升至 85.6%。(b) 峰值工作上下文长度从 16,934 降至 9,634 个 token，接近 8,000 个 token 的惩罚阈值。

Environments. The environment used in our experiments is a modified and harder version of ALFWorld. We make the following modifications to make it more suitable for evaluating long-horizon memory agents: (1) Hidden admissible commands: the default ALFWorld observation includes a list of all valid actions with exact object IDs, which trivializes navigation. We remove this from the observation, requiring the agent to discover valid actions through exploration. (2) Hidden initial observation: the room description containing all location IDs is stripped from the initial observation. The agent must explicitly call the “look” action to observe its surroundings, placing location IDs in the conversation history rather than the fixed system context. (3) Limited look: the “look” action is restricted to once per episode.

环境。 实验所用环境是 ALFWorld 的一个 经过修改且难度更高的版本。 我们做出以下修改，使其更适合评估长时程记忆智能体：(1) 隐藏可执行命令：默认的 ALFWorld 观测包含所有有效动作及精确对象 ID 的列表，使导航变得过于简单。 我们从观测中移除这些内容，要求智能体通过探索发现有效动作。(2) 隐藏初始观测：从初始观测中移除包含所有位置 ID 的房间描述。 智能体必须显式调用“look”动作来观察周围环境，使位置 ID 出现在对话历史中，而非固定的系统上下文中。(3) 限制 look：每个回合只允许执行一次“look”动作。

After the first use, the location IDs can only be recovered through ReadExperience, creating an explicit dependency on the memory retrieval mechanism. (4) Summary truncation: the summary produced by CompressExperience is truncated to 300 tokens, forcing the agent to store detailed information (e.g., object IDs) in structured db_blocks and retrieve them via ReadExperience rather than embedding everything in the summary. The training set contains 3553 tasks. Detailed prompts used can be found in Appendix B.

首次使用后，位置 ID 只能通过 ReadExperience 恢复，从而形成对记忆检索机制的显式依赖。(4) 摘要截断：CompressExperience 生成的摘要会被截断至 300 个 token，迫使智能体将对象 ID 等详细信息存入结构化的 db_blocks，并通过 ReadExperience 检索，而不是把所有内容都嵌入摘要。 训练集包含 3553 个任务。 所用提示的详情见 附录 B。

Training Details. Our implementation is based on the open-sourced LLM RL framework Slime. We use INT4 quantization for inference rollout and quantization-aware training (QAT) for the backward pass, where weights are quantized to INT4 for forward computation but gradients are accumulated in BF16. To mitigate the inference-training mismatch introduced by multi-turn agent interactions, we adopt truncated importance sampling (TIS) with a clip ratio of 2.0 for loss calculation and a token-in-token-out agent loop that avoids re-tokenization between turns. Training uses the Adam optimizer with learning rate $5\times {10}^{-6}$, weight decay 0.1, and a KL divergence penalty (${\lambda }_{\text{KL}}=0.001$) against the frozen reference model.

训练细节。 我们的实现基于开源 LLM 强化学习框架 Slime。 我们在推理轨迹采样中使用 INT4 量化，并在反向传播中采用量化感知训练（QAT）：前向计算时将权重量化为 INT4，但以 BF16 累积梯度。 为缓解多轮智能体交互引入的推理与训练不匹配，我们在损失计算中采用裁剪比为 2.0 的截断重要性采样（TIS），并使用直接输入和输出 token 的智能体循环，避免在轮次之间重新分词。 训练采用 Adam 优化器，学习率为 $5\times {10}^{-6}$，权重衰减为 0.1，并使用相对于冻结参考模型的 KL 散度惩罚（${\lambda }_{\text{KL}}=0.001$）。

The context window size and the context penalty threshold used in training are 32K and 8K, respectively. The batch size is 32 (i.e., 32 prompts for each rollout step). The group size for GRPO is 8 (i.e., 8 rollout samples per prompt).

训练所用的上下文窗口大小和上下文惩罚阈值分别为 32K 和 8K。 批次大小为 32，即每个轨迹采样步骤使用 32 个提示。 GRPO 的组大小为 8，即每个提示采样 8 条轨迹。

5.2. Effectiveness

We evaluate the effectiveness of MemexRL by comparing the Memex agent with and without RL training on the modified ALFWorld benchmark. The training dynamics already show that the proposed learning framework is effective. As shown in Figure 2, the rollout task success rate increases from approximately 20% to over 90% during training. At the same time, Figure 3 shows that the total penalty improves from around $-0.4$ to approximately $-0.1$. These trends indicate that the agent not only becomes substantially better at completing tasks, but also learns to manage its working context more effectively through the memory actions provided by Memex.

我们在修改后的 ALFWorld 基准上，对比经过与未经过 RL 训练的 Memex 智能体，以评估 MemexRL 的有效性。 训练动态已表明，所提出的学习框架是有效的。 如 图2 所示，训练过程中，采样轨迹的任务成功率从约 20% 提升至 90% 以上。 与此同时，图3 显示总惩罚从约 $-0.4$ 改善至约 $-0.1$。 这些趋势表明，智能体不仅在完成任务方面取得了显著进步，也学会了通过 Memex 提供的记忆动作更有效地管理工作上下文。

The final evaluation results in Figure 4 further confirm this improvement. MemexRL increases the task success rate from 24.22% to 85.61%, while reducing the peak working context length from 16934.46 to 9634.47 tokens. This corresponds to more than a $3.5\times$ improvement in task success together with an approximately 43% reduction in peak working context. These results show that the gain from MemexRL is not simply due to more aggressive compression.

图4 中的最终评估结果进一步证实了这一改进。 MemexRL 将任务成功率从 24.22% 提升至 85.61%，同时将峰值工作上下文长度从 16934.46 降至 9634.47 个 token。 这相当于任务成功率提升至原来的 $3.5$ 倍以上，同时峰值工作上下文缩减约 43%。 这些结果表明，MemexRL 的收益并不只是来自更激进的压缩。

Instead, RL teaches the agent how to use indexed experience memory in a way that better supports downstream reasoning and decision making under a constrained context budget. Note that the context penalty threshold used in training is 8000. The results show that the peak working context length after MemexRL training is close to the threshold.

相反，RL 教会智能体如何使用索引经验记忆，以便在受限的上下文预算下更好地支持下游推理和决策。 注意，训练中使用的上下文惩罚阈值为 8000。 结果显示，MemexRL 训练后的峰值工作上下文长度接近该阈值。

Figure 5 provides additional insight into the learned memory behavior. After training, the mean number of CompressExperience calls per episode decreases from about 6.5 to about 3, while the mean number of ReadExperience calls increases from about 1 to around 6–7. This behavioral shift is important. It suggests that MemexRL does not merely encourage the agent to compress history more frequently.

图5 为学得的记忆行为提供了进一步洞察。 训练后，每个回合的 CompressExperience 平均调用次数从约 6.5 次降至约 3 次，而 ReadExperience 的平均调用次数从约 1 次增至约 6–7 次。 这一行为变化很重要。 它说明 MemexRL 并不只是鼓励智能体更频繁地压缩历史。

Rather, the learned policy compresses more selectively and increasingly relies on explicit retrieval from the external experience store when previously observed evidence becomes relevant again. In other words, RL shifts the agent from repeatedly rewriting context toward building a reusable indexed memory that can be dereferenced precisely when needed. This is exactly the intended operating mode of Memex, and it explains why the trained agent achieves substantially better long-horizon performance with a significantly smaller working context.

相反，学得的策略更有选择地进行压缩，并在先前观察到的证据再次变得相关时，越来越依赖从外部经验存储中进行显式检索。 换言之，RL 使智能体从反复重写上下文，转向构建可复用的索引记忆，并在需要时精确解引用。 这正是 Memex 预期的运行方式，也解释了为何经过训练的智能体能够以显著更小的工作上下文，获得大幅提高的长时程任务性能。

(a) 平均压缩次数

(b) 平均检索次数

图5：训练过程中在评估集上的记忆工具使用情况。(a) 随着智能体更高效地完成任务，压缩次数从 6.5 降至 3。(b) 检索次数从 1 增至 6–7，表明 RL 强化而非抑制了检索行为。

6. Conclusion

We presented Memex, a long-horizon tool-using agent that operates under fixed context windows by separating a compact in-context indexed summary from full-fidelity artifacts stored in an external experience database under stable indices. This Indexed Experience Memory turns long trajectories into a pointer-heavy workflow, enabling the agent to keep actionable state in context while dereferencing exact evidence only when needed. We further introduced MemexRL, which learns both compression and retrieval behaviors with reward shaping tailored to indexed memory usage and segmented trajectory processing for episodes with multiple compressions. Across challenging long-horizon tasks, Memex improves task success while using a significant smaller working context. These results suggest that learning how to summarize, index, and dereference experience is a complementary scaling axis for building more persistent and reliable LLM agents.

我们提出了 Memex，这是一种在固定上下文窗口下运行的长时程工具使用智能体，将紧凑的上下文内 索引摘要 与以稳定索引存储于外部经验数据库中的完整保真产物分离。 这种 索引经验记忆 将长轨迹转化为以指针为主的工作流，使智能体能够在上下文中保留可据以行动的状态，同时仅在需要时解引用确切证据。 我们进一步提出 MemexRL，通过针对索引记忆使用的奖励塑形，以及面向多次压缩回合的分段轨迹处理，同时学习压缩与检索行为。 在具有挑战性的长时程任务中，Memex 提高了任务成功率，同时使用显著更小的工作上下文。 这些结果表明，学习如何摘要、索引和解引用经验，为构建更持久、更可靠的 LLM 智能体提供了一条互补的扩展方向。