---

MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents

MemoryRLPPO310+160+ICLR 2026CCF-A新加坡-麻省理工学院研究与技术联盟

Zhou Z, Qu A, Wu Z, et al. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents. ICLR 2026.

https://github.com/MIT-MI/MEM1

---

MEM1：学习协同记忆与推理以构建高效长程智能体

Abstract

Modern language agents often need to solve long-horizon tasks requiring multiple turns of interactions with the environment, where they retrieve external information, adapt to observations, and answer interdependent queries. Yet, most LLM systems rely on full-context prompting, appending all past turns regardless of their relevance. This leads to unbounded memory growth, increased computational costs, and degraded reasoning performance on out-of-distribution input lengths due to LLM forgetting the context. We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with nearly near constant context size when solving long-horizon tasks. At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning. Leveraging reinforcement learning (RL) and rollout trajectory truncation, we train a MEM1 agent to develop internal states that integrate prior memory with new observations from the environment while strategically discarding irrelevant or redundant information. Experiments across three domains, including internal retrieval QA, open-domain web QA, and multi-turn web shopping, show that MEM1-7B improves performance by $3.5\times$ while reducing memory usage by $3.7\times$ compared to Qwen2.5-14B-Instruct on an augmented multi-hop QA dataset with $16$ objectives in each task, and generalizes beyond the training horizon. Our results demonstrate the promise of reasoning-driven memory consolidation as a scalable alternative to existing solutions for training long-horizon task-solving agents with multiple interactions, where both efficiency and performance are optimized.

现代语言智能体常常需要解决需要与环境进行多轮交互的长程任务，在此过程中检索外部信息、适应观测结果，并回答相互依赖的问题。 然而，大多数 LLM 系统仍依赖完整上下文提示，不管历史轮次是否相关，都会把所有过去交互追加到提示中。 这会导致记忆无界增长、计算成本增加，并由于 LLM 遗忘上下文而使模型在分布外输入长度上推理性能下降。 本文提出 MEM1，这是一个端到端强化学习框架，使智能体在解决长程任务时能够以近似常数的上下文大小运行。 在每一轮中，MEM1 会更新一个紧凑的共享内部状态，同时支持记忆整合与推理。 借助 reinforcement learning (RL) 和 rollout trajectory truncation，我们训练 MEM1 agent 形成内部状态，把先前记忆与来自环境的新观测整合起来，同时有策略地丢弃无关或冗余信息。 在内部检索 QA、开放域 Web QA 和多轮 Web 购物三个领域的实验表明，在每个任务含 $16$ 个 objectives 的增强 multi-hop QA 数据集上，相比 Qwen2.5-14B-Instruct，MEM1-7B 将性能提升 $3.5\times$，同时将记忆使用降低 $3.7\times$，并且能够泛化到训练长度之外。 这些结果说明，由推理驱动的记忆整合可以作为训练具有多次交互的长程任务求解智能体的可扩展替代方案，同时优化效率与性能。

1. Introduction

Large language models (LLMs) have shown remarkable performance in single-turn tasks such as question answering, summarization, and code generation. However, emerging real-world applications increasingly operate over multiple turns--searching documents, interacting with environments, and making decisions based on evolving external information. Examples include AI search agents such as Perplexity AI that automate complex tasks by iteratively gathering information, and web-navigation agents such as OpenManus and BrowserUse, which must complete goals across dozens of interactive turns.

大型语言模型（LLM）已经在问答、摘要和代码生成等单轮任务中展现出显著表现。 然而，新兴真实应用越来越多地运行在多轮场景中：搜索文档、与环境交互，并基于不断变化的外部信息做出决策。 例如 Perplexity AI 这类 AI search agents 会通过迭代收集信息来自动完成复杂任务；OpenManus 和 BrowserUse 这类网页导航智能体则需要在数十轮交互中完成目标。

Unlike traditional tasks where the input is static or self-contained, long-horizon settings often involve answering a sequence of related questions, requiring the agent to continuously retrieve new information, revise beliefs, and adapt to evolving contexts over time. For instance, consider a research assistant tasked with "What is the evidence for X?". Subsequent queries like "Who published it?" require further information retrieval, while "Is the source credible?" calls for self-reflection and assessment. Each query builds on the previously collected and accumulated information. Similarly, a shopping assistant may be first asked "Which product is cheapest?", then "What are its reviews?", and "Is it compatible with my device?". These interactions span multiple turns, featuring evolving contexts and compound reasoning.

传统任务中的输入通常是静态或自包含的，而长程场景往往涉及一系列相关问题，要求智能体持续检索新信息、修正信念，并随着上下文变化不断适应。 例如，一个研究助手可能被要求回答“X 的证据是什么？”。 后续问题如“是谁发表的？”需要进一步检索信息，而“这个来源可靠吗？”则需要自我反思和评估。 每个问题都建立在此前已经收集并积累的信息之上。 类似地，一个购物助手可能先被问到“哪个产品最便宜？”，随后又被问到“它的评价如何？”以及“它和我的设备兼容吗？”。 这些交互跨越多个轮次，具有持续演化的上下文和复合推理。

In long-horizon systems, a common strategy is to append all past observations, actions, and thoughts to the context at each step. This creates three challenges. (1) Growing inference cost and memory usage. Transformer-based LLMs typically incur $O(N^2)$ compute cost (or $O(N)$ with Key-Value caching) and $O(N)$ memory usage as the context length $N$ increases. Consequently, deploying these models requires reserving large GPU memory on modern inference frameworks to accommodate the growing context, often leading to significant wastage of computing resources. (2) Generalization limits beyond the training horizons. Contexts longer than those seen during training push the model out-of-distribution, reducing its ability to reason reliably. (3) Overloaded context and forgetting. Redundant or irrelevant content that grows with agentic interaction dilutes attention and makes the model prone to forgetting important details, even when they remain technically available in the context.

在长程系统中，一种常见策略是在每一步都把所有过去观测、动作和思考追加到上下文中。 这会产生三个挑战。 (1) 推理成本和记忆使用增长。 随着上下文长度 $N$ 增加，基于 Transformer 的 LLM 通常会产生 $O(N^2)$ 的计算成本（或在 Key-Value caching 下为 $O(N)$）以及 $O(N)$ 的记忆使用。 因此，部署这些模型时需要在现代推理框架上预留大量 GPU memory 来容纳不断增长的上下文，往往导致显著的计算资源浪费。 (2) 超出训练长度后的泛化限制。 长于训练时所见长度的上下文会把模型推到分布外，从而降低其可靠推理能力。 (3) 过载上下文和遗忘。 随着智能体交互增长的冗余或无关内容会稀释注意力，使模型更容易遗忘重要细节，即使这些细节从技术上仍在上下文中。

Recent progress in long-context modeling largely targets static inputs (e.g., long documents) and does not address multi-turn interaction with external environments. Some other approaches introduce external memory modules (e.g., summarizers or retrievers), but these are typically trained separately and cannot be optimized end-to-end with the agent's policy. This also introduces additional engineering overhead, as engineers must manage and integrate two separate models. Meanwhile, existing works on tool-using agent systems trained with reinforcement learning leave memory management unsolved, letting the prompt length grow unboundedly. A natural question is raised: Can a language model learn to consolidate its memory as part of its reasoning process so that it retains only what is essential for solving the task?

长上下文建模的近期进展主要针对静态输入（例如长文档），并没有解决与外部环境进行多轮交互的问题。 另一些方法引入外部记忆模块（例如 summarizer 或 retriever），但这些模块通常单独训练，无法与智能体策略进行端到端优化。 这还会引入额外工程开销，因为工程师必须管理并集成两个独立模型。 与此同时，已有的使用工具并通过强化学习训练的智能体系统仍未解决记忆管理问题，任由提示长度无界增长。 这自然引出一个问题：语言模型能否把记忆整合作为其推理过程的一部分来学习，从而只保留解决任务所必需的信息？

Motivated by this question, we present MEM1: Memory-Efficient Mechanism via learning 1-step integrated reasoning and consolidation--a method for training LLM agents that maintain nearly constant memory usage across arbitrarily long horizons.

受这一问题启发，我们提出 MEM1：Memory-Efficient Mechanism via learning 1-step integrated reasoning and consolidation，即通过学习一步式集成推理与整合来实现记忆高效机制，这是一种训练 LLM 智能体的方法，使其能在任意长程任务中保持近似常数的记忆使用。

At each turn, the model updates a consolidated state composed of prior memory and newly obtained information. This consolidated state becomes the agent's only retained memory, allowing all observations obtained via external tool use to be discarded after use, which prevents prompt expansion altogether (illustrated later in Section 4.2). A key insight of our method is that inference-time reasoning serves two purposes: while reasoning about the current query, the model also extracts and stores the essential information it needs for the future. By unifying reasoning and memory consolidation, MEM1 enables the agent to both reason and remember within a shared representational space, without requiring extra modules or architectural changes.

在每一轮中，模型都会更新一个由先前记忆和新获得信息组成的整合状态。 这个整合状态成为智能体唯一保留的记忆，使所有通过外部工具获得的观测都能在使用后被丢弃，从而彻底避免提示膨胀（稍后在第 4.2 节说明）。 本文方法的关键洞见是，推理时 reasoning 有两个目的：模型在推理当前查询的同时，也会提取并存储未来所需的关键信息。 通过统一推理和记忆整合，MEM1 使智能体能够在共享表征空间中同时推理和记忆，而不需要额外模块或架构改动。

We train this behavior end-to-end with reinforcement learning (RL), optimizing for task success via verifiable rewards. Although not explicitly optimized for memory efficiency through reward signals, the agent learns to manage memory as part of its policy, resulting in near-constant memory usage across long horizons. Additionally, we notice that current training and evaluation environments predominantly focus on single-objective tasks, limiting their ability to fully prepare agents for realistic, long-horizon scenarios that inherently involve multiple sequential objectives. To address this challenge, we introduce a scalable task augmentation approach, transforming existing single-objective QA datasets into complex multi-objective tasks through compositions of $N$ multi-hop questions. This approach enables us to repurpose standard benchmarks in our community to more effectively train and evaluate agents on long-horizon reasoning, an increasingly important capability in real-world applications.

我们使用强化学习（RL）端到端训练这种行为，并通过可验证奖励来优化任务成功率。 虽然奖励信号并没有显式优化记忆效率，但智能体会把记忆管理学习为其策略的一部分，从而在长程任务中实现近似常数记忆使用。 此外，我们注意到当前训练和评估环境主要关注单目标任务，这限制了它们为真实长程场景充分准备智能体的能力，而这些场景本质上涉及多个连续目标。 为了解决这一挑战，我们提出一种可扩展的任务增强方法：通过组合 $N$ 个多跳问题，把已有单目标 QA 数据集转换为复杂多目标任务。 这种方法使我们能够重新利用社区中的标准 benchmark，更有效地训练和评估智能体的长程推理能力；这在真实应用中是一项越来越重要的能力。

We empirically evaluate MEM1 on (i) a multi-turn information retrieval task, where standard single-objective QA datasets are augmented into multi-objective settings by composing $N$ multi-hop questions, and (ii) the WebShop environment, which requires the agent to perform multiple steps of interaction with a textual web environment. Across these diverse settings, MEM1 consistently matches or exceeds the performance of leading baselines while achieving efficiency gains of up to $3.5\times$ in memory usage. Moreover, agents trained on our $2$-objective augmented tasks generalize robustly to much harder cases with up to $16$ sequential objectives. At this extreme, MEM1 not only outperforms all baselines in accuracy but also reduces peak memory usage by $1.27\times$ and accelerates inference by $1.78\times$ relative to the strongest uncollapsed baseline.

我们在两类环境中实证评估 MEM1：(i) 多轮信息检索任务，其中标准单目标 QA 数据集通过组合 $N$ 个 multi-hop questions 被增强为 multi-objective settings；(ii) WebShop 环境，其中智能体需要与文本网页环境进行多步交互。 在这些多样设置中，MEM1 始终达到或超过强基线性能，同时在记忆使用上实现最高 $3.5\times$ 的效率提升。 此外，在我们 $2$-objective 增强任务上训练的智能体，可以稳健泛化到最多 $16$ 个 sequential objectives 的更难场景。 在这一极端设置下，MEM1 不仅在准确率上超过所有基线，相比最强的未崩溃基线还将 peak memory usage 降低 $1.27\times$，并将 inference 加速 $1.78\times$。

2. Related Work

2.1. LLM Agents in Multi-Turn Environment

LLM-based agents have evolved from handling single-turn queries to serving as autonomous agents capable of multi-turn interactions such as web navigation and complex research. To enable such capabilities, Yao et al. introduced the ReAct (i.e., Reason + Act) framework, which enhances LLMs' ability to interact with external environments by interleaving reasoning and action. Building on this reasoning-acting prompting paradigm, subsequent works have explored ways to improve agent performance through natural language feedback, enabling iterative refinement. Recently, inference-time scaling has emerged as a promising direction for enabling complex reasoning, with prior research incorporating evaluators (e.g., verifier, reward model) or world models. In addition, there are two major lines of training approaches: (1) behavior cloning (BC), which involves imitating expert trajectories to guide agent behavior by supervised fine-tuning (SFT), and (2) reinforcement learning (RL), which optimizes agent policies by incentivizing desirable outcomes through rewards. These methods aim to align the agents' behaviors with task objectives, enabling more robust and generalizable performance.

基于 LLM 的智能体已经从处理单轮查询，发展为能够进行网页导航和复杂研究等多轮交互的自主智能体。 为了支持这些能力，Yao et al. 提出了 ReAct（即 Reason + Act）框架，该框架通过交错推理与行动来增强 LLM 与外部环境交互的能力。 在这种 reasoning-acting 提示范式基础上，后续工作探索了利用自然语言反馈改进智能体表现的方法，使其能够迭代修正。 近期，推理时扩展成为支持复杂推理的一个有前景方向，已有研究引入 evaluator（例如 verifier、reward model）或 world model。 此外，训练方法主要有两条路线：(1) behavior cloning (BC)，即通过监督微调（SFT）模仿专家轨迹以引导智能体行为；(2) reinforcement learning (RL)，即通过奖励激励期望结果来优化智能体策略。 这些方法旨在让智能体行为与任务目标对齐，从而获得更稳健、更具泛化性的表现。

2.2. Context Management for LLM Agents

A widely adopted approach to context management in LLM-based agent systems involves appending all prior information, such as observations, intermediate thoughts, and actions, into the prompt at each interaction turn. While this method is straightforward and effective when the number of interactions required is small, it results in unbounded context growth, leading to linearly scaled inference memory. Moreover, long contexts often contain irrelevant or redundant information, which impairs the model's reasoning capabilities. To mitigate these issues, recent studies have proposed external memory frameworks, including retrieval-augmented generation and summarization modules and hierarchical working memories. However, these methods are typically applied independently of the agent's policy, creating a disconnect between memory and the reasoning process. In addition, managing and integrating such modules often incurs extra computational overhead and system complexity. Despite these advancements, many RL approaches for training LLM agents still rely on accumulating the full interaction history as memory, leaving memory management during training an underexplored area. In this work, we seek to bridge this gap by tightly integrating memory with the agent's reasoning process, thereby enabling more efficient and context-aware decision-making.

LLM 智能体系统中一种常见的上下文管理方法，是在每轮交互中把所有先前信息追加到提示里，包括观测、中间思考和动作。 当所需交互轮数较少时，这种方法简单且有效，但它会导致上下文无界增长，并使推理记忆线性扩张。 此外，长上下文通常包含无关或冗余信息，会损害模型的推理能力。 为了缓解这些问题，近期研究提出了外部记忆框架，包括检索增强生成、摘要模块和层次化工作记忆。 然而，这些方法通常独立于智能体策略应用，从而造成记忆与推理过程之间的脱节。 此外，管理和集成这类模块往往会带来额外计算开销和系统复杂度。 尽管已有这些进展，许多用于训练 LLM 智能体的 RL 方法仍然依赖于把完整交互历史作为记忆来累积，使得训练期间的记忆管理仍是一个探索不足的领域。 在本文中，我们希望通过把记忆与智能体推理过程紧密整合来弥合这一缺口，从而实现更高效、更具上下文感知能力的决策。

3. MEM1

Complex reasoning tasks often require an iterative process of information gathering and synthesis, as seen in applications such as deep search and web-based agents. We consider an interactive agent operating in a multi-turn environment with vocabulary space $V$. The generation process of an agent is defined as a Markov Decision Process (MDP) parameterized by $(\mathcal{S},\mathcal{A},\pi ,r)$, where $\mathcal{A}$ represents the action space, $\mathcal{S}$ represents the state space, $\pi :\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ is the transition distribution (i.e., the policy), and $r:\mathcal{S}\to \mathbb{R}$ is the reward function. A trajectory generated by a policy $\pi$ is denoted with $\tau :=\{(a_t,s_t)\mid t\in \{1,2,\dots ,|\tau |\}\}$. At each step $t$, the agent receives an observation $O_t\in {\mathcal{A}}^{+}$ (from external tools, APIs, or the environment), maintains an internal state $S_t\in {\mathcal{A}}^{+}$ (reasoning history and memory), and produces an action $A_t\in {\mathcal{A}}^{+}$ (e.g., answering a question or issuing a query), where ${\mathcal{A}}^{+}$ represents the set of sequences of $a\in \mathcal{A}$. The agent's goal is to maximize task success across long-horizon trajectories while keeping the retained context bounded.

复杂推理任务通常需要一个迭代的信息收集与综合过程，这可见于 deep search 和基于 Web 的智能体等应用。 我们考虑一个在多轮环境中运行、词表空间为 $V$ 的交互式智能体。 智能体的生成过程被定义为一个由 $(\mathcal{S},\mathcal{A},\pi ,r)$ 参数化的马尔可夫决策过程（MDP），其中 $\mathcal{A}$ 表示动作空间，$\mathcal{S}$ 表示状态空间，$\pi :\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ 是转移分布（即策略），$r:\mathcal{S}\to \mathbb{R}$ 是奖励函数。 由策略 $\pi$ 生成的轨迹记作 $\tau :=\{(a_t,s_t)\mid t\in \{1,2,\dots ,|\tau |\}\}$。 在每一步 $t$，智能体接收一个观测 $O_t\in {\mathcal{A}}^{+}$（来自外部工具、API 或环境），维护一个内部状态 $S_t\in {\mathcal{A}}^{+}$（推理历史与记忆），并产生一个动作 $A_t\in {\mathcal{A}}^{+}$（例如回答问题或发出查询），其中 ${\mathcal{A}}^{+}$ 表示 $a\in \mathcal{A}$ 的序列集合。 智能体的目标是在保持保留上下文有界的同时，最大化长程轨迹上的任务成功率。

$$\underset{\theta }{\mathrm{argmax}}{\mathbb{E}}_{Q\in \mathcal{Q},\tau \sim {\pi }_{\theta ,Q}}[\sum _{(a_t,s_t)\in \tau }r(s_t)].$$

Formally, let $r:V\to \mathbb{R}$ be the reward function. The learning problem is a maximization over the model parameters $\theta$. In this formulation, $\tau =(S_i,A_i,O_i{)}_{i=1}^{n-1}\cap (S_n,A_n)$ is a trajectory sampled from policy ${\pi }_{\theta ,Q}$ with $n$ turns and $Q$ refers to the set of questions. In this work, we primarily consider tasks with verifiable rewards (i.e., $r$ is a rule-based mapping). A long, multi-turn reasoning task is characterized by a large $n$, requiring the agent to iteratively perform a long series of searches and reasoning to derive the answer $A_n$.

形式化地，令 $r:V\to \mathbb{R}$ 为奖励函数。 学习问题是在模型参数 $\theta$ 上进行最大化。 在这一表述中，$\tau =(S_i,A_i,O_i{)}_{i=1}^{n-1}\cap (S_n,A_n)$ 是从策略 ${\pi }_{\theta ,Q}$ 采样得到、包含 $n$ 轮的轨迹，$Q$ 指问题集合。 在本文中，我们主要考虑具有可验证奖励的任务（即 $r$ 是基于规则的映射）。 一个长程多轮推理任务由较大的 $n$ 表征，要求智能体迭代执行一长串搜索和推理，以推导出答案 $A_n$。

图1：MEM1 的训练 pipeline、上下文演化和 2D attention mask。上方为训练 MEM1 的 RL 流程；左下为 MEM1 中上下文的演化，旧的内部状态 $S$、query/answer $A$ 和外部信息 $O$ 会随着新状态进入上下文而在 rollout 中被清除；右下为 objective computation 阶段使用的 2D attention mask。该 mask 在 forward pass 中用于计算 actor model 的 action log-probabilities 和 critic model 的 state value estimates；policy update 阶段再应用 information mask，以遮蔽并非由模型自身生成的 $O$ tokens。

3.1. Memory as Part of Reasoning

To achieve a constant memory, MEM1 is particularly trained to iteratively refine its understanding by processing new information in conjunction with a consolidation of its prior state. At each turn $i$, the agent produces a new $S_i$, which summarizes past information and reasons about subsequent actions. Following this, the agent generates an action $A_t$--a subsequent query or the answer if a direct response is warranted. If the agent issues a query, the corresponding feedback from the environment $O_i$ is appended to the trajectory. At the next turn, $i+1$, the agent consolidates the tuple $(S_i,A_i,O_i)$ into a new $S_{i+1}$, which serves as the basis for further interactions. After each turn, $(S_i,A_i,O_i)$ is pruned from the context, effectively compressing memory and preventing prompt bloat. Figure 1 (bottom left) illustrates the evolution of the model's context over time. At each turn, the agent retains at most two $S$'s, two $A$'s, and one $O$, ensuring bounded and efficient memory usage. The detailed rollout algorithm is in Algorithm 1 of Appendix B.5.

为了实现常数记忆，MEM1 被专门训练为通过结合新信息与先前状态整合，来迭代修正自己的理解。 在每一轮 $i$，智能体会产生新的 $S_i$，用于总结过去信息并推理后续动作。 随后，智能体会生成动作 $A_t$，也就是后续查询，或者在可以直接回应时生成答案。 如果智能体发出查询，来自环境的对应反馈 $O_i$ 会被追加到轨迹中。 在下一轮 $i+1$，智能体会将元组 $(S_i,A_i,O_i)$ 整合成新的 $S_{i+1}$，作为后续交互的基础。 每轮之后，$(S_i,A_i,O_i)$ 会从上下文中被剪除，从而有效压缩记忆并防止提示膨胀。 图1（左下）展示了模型上下文随时间演化的过程。 在每一轮，智能体最多保留两个 $S$、两个 $A$ 和一个 $O$，从而确保记忆有界且高效。 详细的 rollout 算法见附录 B.5 的 Algorithm 1。

RL offers a powerful mechanism for shaping agent behavior through reward signals. In MEM1, we leverage this framework to incentivize effective state consolidation by designing environments in which the agent is rewarded only when it strategically retains and integrates useful information. Specifically, we construct tasks that require numerous interactions with the environment to arrive at a correct answer (see Section 3.3). Success depends on the agent's ability to rely on information collected along the inference path. At each turn, we prune the agent's context to retain only the most recent internal state $S$, forcing the agent to perform memory consolidation as part of its reasoning process. Without access to full historical context, the agent must learn to preserve and update relevant knowledge internally in order to reap the reward. This learning procedure mirrors how humans cultivate memorization skills through structured tasks such as Sudoku or crosswords, where success hinges on selectively attending to key information and building upon it. Over time, such tasks help individuals develop cognitive strategies that jointly support efficient memorization and reasoning, similar to our RL method for training MEM1.

RL 通过奖励信号为塑造智能体行为提供了强有力的机制。 在 MEM1 中，我们利用这一框架来激励有效的状态整合：我们设计的环境只有在智能体有策略地保留并整合有用信息时才给予奖励。 具体而言，我们构造了需要与环境进行大量交互才能得到正确答案的任务（见第 3.3 节）。 成功取决于智能体依赖推理路径中所收集信息的能力。 在每一轮，我们都会剪除智能体上下文，只保留最近的内部状态 $S$，迫使智能体把记忆整合作为其推理过程的一部分。 由于无法访问完整历史上下文，智能体必须学会在内部保留并更新相关知识，以便获得奖励。 这一学习过程类似于人类通过 Sudoku 或 crossword 等结构化任务培养记忆技能；在这些任务中，成功依赖于选择性关注关键信息并在其基础上继续构建。 随着时间推移，这类任务帮助个体形成同时支持高效记忆与推理的认知策略，这与我们训练 MEM1 的 RL 方法相似。

3.2. Masked Trajectory for Policy Optimization

Popular RL algorithms update the policy with policy gradient, which requires the calculation of ${\mathrm{\nabla }}_{\theta }{\pi }_{\theta ,Q}(a_t,s_t)$. For LLM, ${\pi }_{\theta ,Q}(a_t,s_t)$ is viewed as the logit of the output $a_t$ of the model, where the input is $s_t$. Existing RL frameworks typically compute the ${\mathrm{\nabla }}_{\theta }{\pi }_{\theta ,Q}(a_t,s_t)$'s for all pairs $(a_t,s_t)\in \tau$ by passing the entire rollout trajectory $\tau$ through the LLM once (i.e., prefilling). However, as MEM1 dynamically consolidates its context, the tokens $a_t$ do not belong to one single trajectory $\tau$. A naive solution is to break each turn into a sub-trajectory ${\tau }_i=(S_i,A_i,O_i)$, where $i$ represents the $i$th interaction turn. However, this approach introduces difficulties (at least implementation-wise) in computing the temporal difference ${\delta }_t=r(s_t)+V(s_{t+1})-V(s_t)$ for the last token in the current sub-trajectory ${\tau }_i$, as $V(s_{t+1})$ is calculated in a separate sub-trajectory ${\tau }_j$, $j\ne i$. Here $V:\mathcal{S}\to \mathbb{R}$ is the value function.

流行的 RL 算法使用 policy gradient 更新策略，这需要计算 ${\mathrm{\nabla }}_{\theta }{\pi }_{\theta ,Q}(a_t,s_t)$。 对于 LLM，${\pi }_{\theta ,Q}(a_t,s_t)$ 被视为模型输出 $a_t$ 的 logit，其中输入为 $s_t$。 现有 RL 框架通常通过将整个 rollout 轨迹 $\tau$ 一次性传入 LLM（即 prefilling），来计算所有 $(a_t,s_t)\in \tau$ 对应的 ${\mathrm{\nabla }}_{\theta }{\pi }_{\theta ,Q}(a_t,s_t)$。 然而，由于 MEM1 会动态整合其上下文，token $a_t$ 并不属于同一条单一轨迹 $\tau$。 一个朴素解法是将每一轮拆成子轨迹 ${\tau }_i=(S_i,A_i,O_i)$，其中 $i$ 表示第 $i$ 个交互轮次。 然而，这种方法会在计算当前子轨迹 ${\tau }_i$ 最后一个 token 的 temporal difference ${\delta }_t=r(s_t)+V(s_{t+1})-V(s_t)$ 时带来困难（至少在实现层面如此），因为 $V(s_{t+1})$ 是在另一个子轨迹 ${\tau }_j$、$j\ne i$ 中计算的。 这里 $V:\mathcal{S}\to \mathbb{R}$ 是 value function。

To overcome this challenge, we introduce a masked trajectory that compresses $\{{\tau }_1,{\tau }_2,\dots ,{\tau }_n\}$ for a task with $n$ turns into a consolidated full trajectory. The full trajectory encodes all information needed for accurate policy learning while respecting MEM1's memory consolidation at each turn. Note that ${\tau }_{\mathrm{full}}$ is a "stitched" trajectory where the $S_i$'s and $A_i$'s do not belong to the same roll-out. As such, to ensure that policy gradients ${\mathrm{\nabla }}_{\theta }{\pi }_{\theta ,Q}(a_t,s_t)$ are correctly computed under this consolidated memory regime, we apply a two-dimensional attention mask across ${\tau }_{\mathrm{full}}$. This mask restricts each token's attention to only the tokens retained in memory at the time that token was generated.

为了克服这一挑战，我们引入 masked trajectory，将一个包含 $n$ 轮的任务中的 $\{{\tau }_1,{\tau }_2,\dots ,{\tau }_n\}$ 压缩为整合后的完整轨迹。 完整轨迹在遵守 MEM1 每轮记忆整合机制的同时，编码了准确策略学习所需的全部信息。 注意，${\tau }_{\mathrm{full}}$ 是一条“拼接”轨迹，其中各个 $S_i$ 和 $A_i$ 并不属于同一次 roll-out。 因此，为了确保在这种整合记忆机制下正确计算 policy gradient ${\mathrm{\nabla }}_{\theta }{\pi }_{\theta ,Q}(a_t,s_t)$，我们在 ${\tau }_{\mathrm{full}}$ 上应用二维 attention mask。 该 mask 限制每个 token 只能关注该 token 生成时仍被保留在记忆中的 token。

$$\begin{array}{rl}{\tau }_{\mathrm{full}}& =({\tau }_1,{\tau }_2,\dots ,{\tau }_n)\\ & =(S_1,A_1,O_1,S_2,A_2,O_2,\dots ,S_n,A_n).\end{array}$$$$\begin{array}{rl}{\mathrm{Attn}}_t& ={\mathbf{1}}_{a\in \{S_{i-1},A_{i-1},O_{i-1},S_i,A_i,O_i\}}\\ & \times {\mathbf{1}}_{a\in \{a_k\mid k\in \{1,2,\dots ,t\}\}}.\end{array}$$$$\begin{array}{rl}{\pi }_{\theta ,Q,{\tau }_i}(a_t,s_t)& ={\pi }_{\theta ,Q,{\tau }_{\mathrm{full}}}(a_t,s_t\times {\mathrm{Attn}}_t).\end{array}$$

Specifically, let the attention mask for the $t$th token in the $i$th turn be ${\mathrm{Attn}}_t$. We can compute the policy for the $i$th turn ${\pi }_{\theta ,Q,{\tau }_i}(a_t,s_t)$ as the policy on ${\tau }_{\mathrm{full}}$ with the masked input, and compute ${\mathrm{\nabla }}_{\theta }{\pi }_{{\tau }_i}$ accordingly. Algorithmically, this can be achieved by first constructing the attention mask $\mathrm{Mask}=({\mathrm{Attn}}_1,\dots ,{\mathrm{Attn}}_T)$ and masking the attention matrix during the transformer forward. Figure 1 (bottom right) shows the masking mechanism that enables stable and accurate policy optimization under MEM1's memory-constrained execution.

具体而言，令第 $i$ 轮中第 $t$ 个 token 的 attention mask 为 ${\mathrm{Attn}}_t$。 我们可以把第 $i$ 轮的策略 ${\pi }_{\theta ,Q,{\tau }_i}(a_t,s_t)$ 计算为 ${\tau }_{\mathrm{full}}$ 上带 masked input 的策略，并据此计算 ${\mathrm{\nabla }}_{\theta }{\pi }_{{\tau }_i}$。 在算法上，可以先构造 attention mask $\mathrm{Mask}=({\mathrm{Attn}}_1,\dots ,{\mathrm{Attn}}_T)$，并在 transformer forward 期间遮蔽 attention matrix。 图1（右下）展示了这一 masking 机制，它使 MEM1 在记忆受限执行方式下能够进行稳定而准确的策略优化。

3.3. Multi-Objective Task Design

Although MEM1 is designed to address the tasks involving multi-turn interaction with the external world, there are limited publicly available datasets that support training for such long-horizon interactive processes. Existing benchmarks, such as HotpotQA, Bamboogle, and 2wiki, are often cited as multi-hop benchmarks, yet they typically involve only two information-seeking steps. Moreover, these datasets are not explicitly structured to support long-horizon interactions that necessitate the agent to manage the memory state.

虽然 MEM1 旨在处理涉及与外部世界进行多轮交互的任务，但公开可用于训练这种长程交互过程的数据集仍然有限。 HotpotQA、Bamboogle 和 2wiki 等已有 benchmark 常被视为 multi-hop benchmark，但它们通常只涉及两个 information-seeking steps。 此外，这些数据集并没有被显式设计成支持需要智能体管理记忆状态的长程交互。

To bridge this gap, we introduce a novel task--multi-objective question answering (QA)--that extends the number of reasoning steps required to solve a problem. Building on existing multi-turn datasets such as HotpotQA and Natural Questions, we interleave multiple questions from the original QA corpus and construct a single composite query that requires answering all constituent sub-questions, shown in Prompt 1 of Appendix B.3. Unlike standard multi-turn QA, this formulation compels the agent to (i) issue multiple search queries, each targeting a distinct sub-question, and (ii) organize the sub-answers into a coherent final response. The augmented dataset inherits the multi-turn retrieval tasks presented in the original HotpotQA. Additionally, to test the agent's long-horizon processing capability, our multi-objective QA combines multiple multi-turn questions into a grand objective and tasks the agent to answer all the questions combined. For instance, two original HotpotQA questions can be combined into one augmented task that asks the agent to answer each question, organize the final answer in <answer> and </answer>, and separate the answer to each question with semicolon.

为了弥合这一缺口，我们提出了一种新任务 multi-objective question answering (QA)，它会增加问题求解所需的推理步骤数。 基于 HotpotQA 和 Natural Questions 等已有多轮数据集，我们交错组合原始 QA 语料中的多个问题，并构造一个单一复合查询，要求回答所有组成子问题，相关内容见附录 B.3 的 Prompt 1。 不同于标准多轮 QA，这种形式迫使智能体：（i）发出多条搜索查询，每条针对一个不同子问题；（ii）将子答案组织成连贯的最终回答。 增强后的数据集继承了原始 HotpotQA 中的多轮检索任务。 此外，为测试智能体的长程处理能力，我们的 multi-objective QA 会将多个多轮问题组合成一个总体目标，并要求智能体回答所有组合后的问题。 例如，两个原始 HotpotQA 问题可以被组合成一个增强任务，要求智能体分别回答每个问题，把最终答案组织在 <answer> 与 </answer> 中，并用分号分隔每个问题的答案。

4. Experiments & Results

We empirically demonstrate the effectiveness of our approach in training the MEM1 agent to perform multi-turn tasks while preserving a near-constant-sized memory state. We evaluate MEM1 against several baselines using a comprehensive set of metrics categorized into accuracy (e.g., Exact Match, F1 score, Environment Reward) and efficiency (e.g., Peak Token Usage, Dependency Length, Inference Time). All MEM1 variants are fine-tuned from the Qwen2.5-7B Base model. We use PPO as the RL algorithm as it computes token-level advantages, bringing stability to the training process. While we also experimented with instruction-tuned and supervised fine-tuned models using curated high-quality trajectories, reinforcement learning from the base model consistently yielded the best performance and generalization.

我们通过实验验证本文方法在训练 MEM1 智能体执行多轮任务、同时保持近似常数大小记忆状态方面的有效性。 我们使用一组综合指标将 MEM1 与多个基线比较，这些指标分为准确性（例如 Exact Match、F1 score、Environment Reward）和效率（例如 Peak Token Usage、Dependency Length、Inference Time）两类。 所有 MEM1 变体都从 Qwen2.5-7B Base 模型微调而来。 我们使用 PPO 作为 RL 算法，因为它可以计算逐 token advantage，从而为训练过程带来稳定性。 虽然我们也实验了 instruction-tuned 模型，以及使用精选高质量轨迹进行 supervised fine-tuned 的模型，但从 base model 出发进行 reinforcement learning 始终带来最佳性能和泛化。

Our experiments are conducted in two standard environments, each reflecting real-world scenarios that require multi-turn agent interactions. The first environment is question answering with retrieval-augmented generation (RAG), where the agent must answer queries by retrieving relevant information from an external knowledge store (either a database or an online search engine). We trained on RAG with a local database (i.e., Wikipedia Corpus) and evaluated on tasks involving open web browsing. For QA, following Section 3.3, we construct multi-objective tasks and tested the model performance on tasks with more questions than seen in the training. The second environment is WebShop navigation, where the agent assists users in online shopping by browsing a website and selecting items based on natural language descriptions. This task requires the agent to iteratively read page content and make navigation decisions, following protocols similar to those in WebGPT.

我们的实验在两个标准环境中进行，每个环境都反映了需要多轮智能体交互的真实场景。 第一个环境是带检索增强生成（RAG）的问答任务，智能体必须从外部知识库（数据库或在线搜索引擎）中检索相关信息来回答查询。 我们在带本地数据库（即 Wikipedia Corpus）的 RAG 上训练，并在涉及开放网页浏览的任务上评估。 对于 QA，我们遵循第 3.3 节构造 multi-objective tasks，并在问题数量超过训练时所见数量的任务上测试模型表现。 第二个环境是 WebShop navigation，智能体需要根据自然语言描述浏览购物网站并选择商品，帮助用户完成在线购物。 该任务要求智能体迭代地阅读页面内容并做出导航决策，遵循与 WebGPT 类似的协议。

4.1. Implementation Details

Datasets and evaluation metrics. We train two versions of MEM1 agent for both long-horizon QA and web navigation. For long-horizon QA, we augment the multi-turn QA dataset that mixes data from both HotpotQA and Natural Question to form a multi-objective composite tasks. During training, we use 2-objective task only and test the agent's performance on tasks with more objectives. For the web agent, we use the WebShop environment, which also produces a reward during training. For all datasets, the train-test split follows the original papers. During RL training, we employ the exact match (EM) metric for QA tasks (details in Appendix B.4.1) and the environment reward for WebShop. To evaluate the effectiveness of various approaches, we measure the EM and F1 score for QA tasks and final reward for the WebShop environment. To evaluate efficiency, we consider the peak token usage, average dependency, and average inference time. The test datasets are obtained from the original papers which consist of out-of-distribution data. The former two metrics measure the memory efficiency, while the latter measures the time efficiency. The detailed definitions of the metrics are in Appendix B.4.1. The prompt and format can be found in Appendix B.3.

数据集和评估指标。 我们为 long-horizon QA 和 web navigation 分别训练两个版本的 MEM1 agent。 对于 long-horizon QA，我们增强了混合 HotpotQA 和 Natural Question 数据的 multi-turn QA 数据集，以形成 multi-objective 复合任务。 训练期间，我们只使用 2-objective task，并在更多 objectives 的任务上测试智能体性能。 对于 web agent，我们使用 WebShop 环境，该环境在训练时也会产生奖励。 对于所有数据集，train-test split 遵循原论文。 在 RL 训练中，我们对 QA 任务使用 exact match (EM) 指标（细节见附录 B.4.1），对 WebShop 使用环境奖励。 为了评估各种方法的有效性，我们对 QA 任务测量 EM 和 F1 score，并对 WebShop 环境测量 final reward。 为了评估效率，我们考虑 peak token usage、average dependency 和 average inference time。 测试数据集取自原论文，其中包含 out-of-distribution 数据。 前两个指标衡量 memory efficiency，后一个指标衡量 time efficiency。 这些指标的详细定义见附录 B.4.1。 Prompt 和 format 见附录 B.3。

Baselines. To evaluate the accuracy and efficiency of MEM1, we compare it against two groups of baselines: (i) prior published methods, and (ii) ablations of our approach. Prior published baselines.

• QA environment: Search-R1, DeepResearcher, and the larger-scale model Qwen2.5-14B-Instruct. Details of Search-R1 and DeepResearcher are provided in Appendix B.4.2.
• WebShop environment: Agent-FLAN, Agent-R, and AgentLM.
• Context compression baseline: A-MEM, which augments an Instruct model with a vector database for memory retrieval.

Ablations of our method.

• Truncation baseline (prompt only): We apply MEM1's agentic truncation prompt template and rollout to an instruct model without RL, isolating the effect of the prompt and rollout design alone. We find that training provides significant performance gains, even though prompt-only rollout already offers some efficiency benefits. Detailed results are presented in Table 1, Table 2, and Table 3.
• SFT baseline: Train a supervised fine-tuned model on trajectories curated from GPT-4o based on MEM1's rollout, enabling comparison with the RL-trained agent. We find that although supervised fine-tuning improves performance, reinforcement learning is much more effective at enabling generalizability. Detailed results are presented in Appendix F.1.
• Memory-Reasoning Coupling: Explicitly separate memory and coupling in MEM1's internal state to investigate the effect of integrating the two in terms of performance and efficiency. We find that integrated memory and reasoning can benefit both performance and efficiency in our evaluated tasks. Detailed results are presented in Appendix F.3.

基线。 为了评估 MEM1 的准确性和效率，我们将其与两组基线比较：（i）已有公开方法；（ii）我们方法的消融。 已有公开基线。

• QA 环境： Search-R1、DeepResearcher 和更大规模模型 Qwen2.5-14B-Instruct。Search-R1 和 DeepResearcher 的细节见附录 B.4.2。
• WebShop 环境： Agent-FLAN、Agent-R 和 AgentLM。
• Context compression baseline： A-MEM，它用 vector database 增强 Instruct model 以进行 memory retrieval。

我们方法的消融。

• Truncation baseline (prompt only)： 我们把 MEM1 的 agentic truncation prompt template 和 rollout 应用于没有 RL 的 instruct model，从而隔离 prompt 和 rollout design 本身的作用。我们发现 training 带来了显著性能增益，尽管 prompt-only rollout 已经提供了一定效率收益。详细结果见表1、表2和表3。
• SFT baseline： 在基于 MEM1 rollout 并由 GPT-4o 筛选的轨迹上训练 supervised fine-tuned model，以便与 RL-trained agent 比较。我们发现，尽管 supervised fine-tuning 会提升性能，reinforcement learning 对实现泛化能力更有效。详细结果见附录 F.1。
• Memory-Reasoning Coupling： 在 MEM1 的 internal state 中显式分离 memory 和 coupling，以研究将二者整合起来对性能和效率的影响。我们发现，integrated memory and reasoning 能够让评估任务中的性能和效率都受益。详细结果见附录 F.3。

Meta info injection. In our agentic pipeline, the agent's context is programmatically truncated at each turn--immediately after it generates a search query or an answer--following the procedure outlined in Section 3. As past context is truncated, the agent may have difficulty determining when to terminate. To address this, we prepend a hint [HINT: YOU HAVE {turns_left} TURNS LEFT] at the beginning of each <info> tag to remind the agent of its remaining turns budget. For all experiments, we set the maximally allowed turns to $6$ for $1$-objective to $4$-objective tasks and $20$ for more difficult tasks to avoid excessively long trajectories.

Meta info injection. 在我们的 agentic pipeline 中，智能体上下文会按照第 3 节所述流程，在每一轮生成 search query 或 answer 后立即由程序截断。 由于过去上下文被截断，智能体可能难以判断何时终止。 为了解决这一点，我们会在每个 <info> 标签开头加入提示 [HINT: YOU HAVE {turns_left} TURNS LEFT]，提醒智能体剩余的轮次预算。 在所有实验中，我们把 $1$-objective 到 $4$-objective 任务的最大允许轮数设为 $6$，更难任务设为 $20$，以避免过长轨迹。

4.2. MEM1 on Multi-Objective Multi-Hop Tasks

One key advantage of MEM1 agents lies in their efficient management of long-horizon interactions with the environment. To demonstrate this, we train our MEM1 agent with a 2-objective augmentation of the QA dataset, and subsequently test it against other models on held-out multi-objective test sets similarly constructed from the original QA corpus. As elaborated in Section 3.3, these multi-objective tasks require substantially more reasoning turns to complete, thus serving as a more demanding benchmark for memory management. As shown in Table 1, MEM1 consistently outperforms other 7B counterparts: when tested on 16-objective task, it achieves over $10\times$ higher EM score, while reducing peak context length by more than $70\mathrm{\%}$ and cutting inference latency by about one-half. This demonstrates that MEM1 not only improves task success on more complex multi-objective settings, but also does so with markedly better efficiency in both memory and runtime.

MEM1 智能体的一个关键优势是能够高效管理与环境之间的长程交互。 为了证明这一点，我们用 QA 数据集的 2-objective 增强版本训练我们的 MEM1 agent，随后在由原始 QA 语料类似构造的 held-out multi-objective test sets 上将其与其他模型比较。 如第 3.3 节所述，这些 multi-objective tasks 需要显著更多推理轮次才能完成，因此是更具挑战性的 memory management benchmark。 如表1所示，MEM1 始终超过其他 7B 对照模型：在 16-objective task 上测试时，它取得超过 $10\times$ 的 EM score，同时将 peak context length 降低超过 $70\mathrm{\%}$，并将 inference latency 缩短约一半。 这表明 MEM1 不仅提升了更复杂 multi-objective settings 下的任务成功率，也以明显更好的 memory 和 runtime efficiency 做到了这一点。

The advantage of MEM1 becomes even more evident in tasks requiring longer-horizon interactive processes. A visualization of the trend in agent performance as the complexity of tasks increases is shown in Figure 2. Figure 2 illustrates the scaling trends of task performance (measured by EM count) and memory efficiency (measured by Peak Token Usage) for MEM1 relative to other models and memory management baselines. As the number of objectives increases, the Peak Token Usage of all other methods and models scales nearly linearly. In contrast, MEM1 maintains an almost constant peak token count with only a slight increase. Notably, while MEM1 initially underperforms Qwen2.5-14B-Instruct, its performance gradually catches up as the number of objectives increases, eventually surpassing the 14B model, which has double the parameter count. As shown in Figure 2, at $1,2,3,4,6$ objectives, MEM1 has close EM scores compared to the 14B model. As the number of objectives continues to increase, MEM1 achieves significantly higher EM scores. In the $16$-objective task, MEM1 achieves over $3\times$ the EM score, while requiring only $27.1\mathrm{\%}$ of the peak tokens and $29.3\mathrm{\%}$ of the total inference time compared to Qwen2.5-14B-Instruct. This efficiency translates to significantly reduced GPU memory requirements and overall computing resource demands.

在需要更长程交互过程的任务中，MEM1 的优势更加明显。 智能体性能随任务复杂度增加而变化的趋势可视化见图2。 图2展示了随着目标数量增加，MEM1 相对于其他模型和记忆管理基线的任务性能（用 EM count 衡量）和 memory efficiency（用 Peak Token Usage 衡量）的 scaling trends。 随着目标数量增加，所有其他方法和模型的 Peak Token Usage 几乎线性增长。 相比之下，MEM1 的 peak token count 几乎保持常数，只略有增加。 值得注意的是，虽然 MEM1 起初弱于 Qwen2.5-14B-Instruct，但随着目标数量增加，其性能逐渐追上，并最终超过参数量翻倍的 14B 模型。 如图2所示，在 $1,2,3,4,6$ 个 objectives 时，MEM1 的 EM scores 与 14B 模型接近。 随着 objectives 数继续增加，MEM1 取得显著更高的 EM scores。 在 $16$-objective task 中，相比 Qwen2.5-14B-Instruct，MEM1 取得超过 $3\times$ 的 EM score，同时只需要 $27.1\mathrm{\%}$ 的 peak tokens 和 $29.3\mathrm{\%}$ 的 total inference time。 这种效率转化为显著降低的 GPU memory requirements 和整体 computing resource demands。

图2：MEM1（在 2-objective QA 上训练）在 multi-objective tasks 中随 objectives 数量变化的性能与效率 scaling。MEM1 在超过其他模型和基线的同时，memory usage 几乎保持常数。注意在 16-objective 时，部分 baseline models 的 context 不再增加，因为其模型性能已经退化（部分 collapsed）。

表1：multi-objective multi-turn QA tasks 上的模型比较。箭头表示期望方向；红色数字表示 collapsed model behavior（极低性能）。truncate 表示使用 MEM1 的 prompt 和 rollout pipeline；A-MEM 表示在 MEM1 prompt 和 rollout pipeline 下使用 A-Mem 的 external memory module；MEM1-QA 表示在 2-objective QA task 上训练的 MEM1。Dependency scores 因空间限制未展示。

Model	2-Objective
	EM ↑	F1 ↑	Peak (×10²) ↓	Time (s) ↓
Qwen2.5-14B-Inst	0.732	0.902	15.6 ± 0.19	5.49 ± 0.16
Qwen2.5-7B-Inst	0.268	0.366	19.6 ± 0.33	4.60 ± 0.08
Qwen2.5-7B-Inst (A-MEM)	0.286	0.371	14.1 ± 0.10	24.6 ± 0.51
Qwen2.5-7B-Inst (truncate)	0.262	0.336	8.28 ± 0.06	5.89 ± 0.16
Search-R1 (original)	0.452	0.531	13.0 ± 0.08	4.09 ± 0.23
Search-R1 (trained on 2-obj task)	0.544	0.646	13.68 ± 0.23	10.60 ± 0.32
Search-R1 (trained on 2-obj task + truncate)	0.446	0.546	6.12 ± 0.04	17.5 ± 0.22
DeepResearcher	0.536	0.650	22.0 ± 0.43	4.01 ± 0.07
MEM1-QA	0.709	0.838	6.40 ± 0.02	6.49 ± 0.07

Model	8-Objective
	EM ↑	F1 ↑	Peak (×10²) ↓	Time (s) ↓
Qwen2.5-14B-Inst	1.55	1.87	44.7 ± 0.37	16.2 ± 0.27
Qwen2.5-7B-Inst	0.87	1.10	49.5 ± 0.40	13.9 ± 0.18
Qwen2.5-7B-Inst (A-MEM)	1.13	1.43	18.6 ± 0.10	53.7 ± 1.26
Qwen2.5-7B-Inst (truncate)	0.97	1.23	11.8 ± 0.10	11.9 ± 0.20
Search-R1 (original)	0.064	0.08	24.7 ± 0.19	4.25 ± 0.16
Search-R1 (trained on 2-obj task)	0.471	0.575	22.48 ± 0.47	19.60 ± 0.50
Search-R1 (trained on 2-obj task + truncate)	0.162	0.204	4.95 ± 0.02	21.5 ± 0.18
DeepResearcher	0.73	0.90	51.8 ± 0.35	11.3 ± 0.14
MEM1-QA	1.87	2.31	8.01 ± 0.06	8.68 ± 0.12

Model	16-Objective
	EM ↑	F1 ↑	Peak (×10²) ↓	Time (s) ↓
Qwen2.5-14B-Inst	0.567	0.703	38.4 ± 0.71	29.7 ± 0.75
Qwen2.5-7B-Inst	0.165	0.213	43.3 ± 0.62	15.5 ± 0.23
Qwen2.5-7B-Inst (A-MEM)	0.730	0.961	18.8 ± 0.14	91.2 ± 2.44
Qwen2.5-7B-Inst (truncate)	0.396	0.497	13.3 ± 0.16	22.1 ± 0.60
Search-R1 (original)	0.009	0.011	20.9 ± 0.03	4.75 ± 0.18
Search-R1 (trained on 2-obj task)	0.520	0.647	24.8 ± 0.57	23.35 ± 0.77
Search-R1 (trained on 2-obj task + truncate)	0.091	0.107	5.28 ± 0.03	24.6 ± 0.22
DeepResearcher	0.071	0.106	48.9 ± 0.66	15.8 ± 0.19
MEM1-QA	1.97	2.39	10.4 ± 0.09	8.70 ± 0.12

4.3. MEM1 on Single-Objective Multi-Hop Tasks

While MEM1 is designed to train agents for very long-horizon tasks, our training method also delivers improved capability with existing multi-hop tasks while achieving much greater efficiency at the same time, all without being explicitly trained on the single-objective versions of these tasks. Note that single-objective tasks also require multiple turns of interaction to produce the desired output. Long-horizon web navigation in WebShop. Beyond QA tasks, we further evaluate the effectiveness of MEM1 in managing long-horizon interactions in the form of web navigation. We show the experimental results in Table 2. Trained in the WebShop environment (see Appendix B.6), MEM1 outperforms other agent training baselines, including Agent-Flan, Agent-R, and AgentLM when utilizing models of similar size. Furthermore, MEM1 achieves remarkable efficiency improvements compared to the best baseline method, AgentLM, featuring a $2.8\times$ improvement in Peak Token Usage, a $1.9\times$ improvement in Dependency, and a $1.5\times$ improvement in Inference Time. MEM1 even surpasses AgentLM-13B, a model with twice the parameter count of our trained model. Additionally, our results indicate that using MEM1 is significantly better than OpenAI's GPT-4o on the WebShop tasks, even when the truncation prompt templates or A-MEM techniques are applied to GPT-4o.

虽然 MEM1 的目标是训练面向超长程任务的智能体，但我们的训练方法在已有 multi-hop tasks 上也能提升能力，同时显著提高效率，而且并没有在这些任务的 single-objective 版本上显式训练。 注意，single-objective tasks 也需要多轮交互才能产生期望输出。 WebShop 中的长程网页导航。 除了 QA 任务，我们进一步评估 MEM1 以 web navigation 形式管理 long-horizon interactions 的有效性。 实验结果见表2。 在 WebShop 环境中训练后（见附录 B.6），MEM1 在使用相似规模模型时超过了其他 agent training baselines，包括 Agent-Flan、Agent-R 和 AgentLM。 此外，相比最佳基线 AgentLM，MEM1 实现了显著效率提升：Peak Token Usage 提升 $2.8\times$，Dependency 提升 $1.9\times$，Inference Time 提升 $1.5\times$。 MEM1 甚至超过了 AgentLM-13B，后者参数量是我们训练模型的两倍。 此外，我们的结果表明，在 WebShop 任务上使用 MEM1 明显优于 OpenAI 的 GPT-4o，即使对 GPT-4o 应用 truncation prompt templates 或 A-MEM 技术也是如此。

表2：WebShop 实验结果。为公平比较，不报告 GPT 的推理时间。Agent-R 分数来自原论文，因为该模型不开源。MEM1-WebShop 表示在 WebShop 环境上训练的 MEM1。

Model	Avg Final Reward ↑	Peak Token (×10³) ↓	Dependency (×10⁶) ↓	Inference Time Per Traj (s) ↓
GPT-4o	25.48	5.30 ± 1.23	3.99 ± 1.16	N/A
GPT-4o (truncate)	13.82	0.99 ± 0.99	0.81 ± 0.23	N/A
GPT-4o (A-MEM)	24.50	1.84 ± 0.06	0.31 ± 0.11	N/A
Qwen2.5-7B-Instruct	18.42	5.64 ± 1.34	3.38 ± 0.89	12.31 ± 1.82
Qwen2.5-14B-Instruct	12.34	5.44 ± 0.92	3.30 ± 0.61	18.17 ± 2.32
Agent-FLAN-7B	40.35	3.37 ± 1.12	2.18 ± 1.62	9.95 ± 6.19
Agent-R-8B	63.91	N/A	N/A	N/A
AgentLM-7B	63.60	2.24 ± 0.40	0.28 ± 0.07	3.91 ± 1.07
AgentLM-13B	70.80	2.36 ± 0.46	0.30 ± 0.08	5.23 ± 1.59
MEM1-WebShop	70.87	0.81 ± 0.10	0.15 ± 0.16	2.61 ± 0.48

For single-objective QA in Wikipedia, Table 3 presents accuracy and efficiency metrics where the agent can make retrieval requests from a Wikipedia datastore via RAG. The MEM1 used in this evaluation is the same as the one detailed in Section 4.2, which is trained solely on a 2-objective task. Overall, MEM1 demonstrates superior efficiency across all three evaluated efficiency metrics, while simultaneously achieving the highest EM score and an F1 score comparable to that of Qwen2.5-14B-Instruct. This improvement in efficiency is attributed to the MEM1 agent's ability to consolidate memory from previous interactions into a compact internal state, which reduces the number of tokens used in the context. We also observe that SFT significantly underperforms RL, highlighting the necessity for RL-based training.

对于 Wikipedia 中的单目标 QA，表3给出了准确性和效率指标；在该环境中，智能体可以通过 RAG 向 Wikipedia 数据库发起检索请求。 本次评估使用的 MEM1 与第 4.2 节详述的模型相同，仅在 2-objective 任务上训练。 总体而言，MEM1 在三个评估的效率指标上都表现更优，同时取得最高 EM 分数，F1 score 也与 Qwen2.5-14B-Instruct 相当。 这种效率提升归因于 MEM1 agent 能够把先前交互中的记忆整合进紧凑内部状态，从而减少上下文中使用的 token 数。 我们还观察到 SFT 明显弱于 RL，这说明基于 RL 的训练是必要的。

Transfer performance to online Web-QA. To validate the transferability and generalizability of the trained MEM1 agent, we test MEM1 trained on RAG-QA in an online web-QA environment, which is unseen by the agent. This environment is similar to the Wiki-QA task in Section 4.3. However, instead of retrieving information from a local Wiki document store, the agent needs to conduct web searches through an API service (e.g., Google Search API) that returns results including titles, snippets, and URLs to answer the QA problems. As shown in Table 3, MEM1 consistently exhibited improved efficiency alongside comparable effectiveness in this unseen setting. The result demonstrates that MEM1 has learned actual reasoning and memory management capability rather than overfitting to the RAG dataset.

迁移到 online Web-QA 的性能。 为了验证已训练 MEM1 agent 的迁移能力和泛化能力，我们在一个智能体未见过的 online web-QA 环境中测试在 RAG-QA 上训练的 MEM1。 该环境类似于第 4.3 节中的 Wiki-QA 任务。 不过，智能体不是从本地 Wiki document store 检索信息，而是需要通过 API 服务（例如 Google Search API）执行网页搜索，该服务会返回标题、摘要片段和 URL，用于回答 QA problems。 如表3所示，MEM1 在这一未见设置中始终展现出更高效率，同时保持相当效果。 该结果表明，MEM1 学到的是真正的 reasoning 和 memory management 能力，而不是过拟合 RAG 数据集。

表3：out-of-distribution single-objective tasks 的跨环境 transfer performance 比较。箭头表示期望方向。SFT 表示用 SFT 训练并应用 MEM1 的 prompt 与 rollout。DeepResearcher 专门在 single-objective Online Web-QA task 上以 F1 score 为优化目标训练，Search-R1 专门在 single-objective Wiki-RAG task 上以 EM 为目标训练。

Environment	System	EM ↑	F1 ↑	Peak Token (×10²) ↓	Dependency (×10⁵) ↓	Inference Time ↓
Wiki RAG	Qwen2.5-7B-Inst (truncate)	0.287	0.382	6.28 ± 0.05	1.65 ± 0.04	2.26 ± 0.04
	Qwen2.5-7B-Inst (A-MEM)	0.246	0.373	8.47 ± 0.12	0.92 ± 0.03	11.2 ± 0.40
	Qwen2.5-7B-Inst	0.269	0.390	9.32 ± 0.19	1.17 ± 0.04	2.31 ± 0.04
	Qwen2.5-14B-Inst	0.422	0.534	8.89 ± 0.21	2.22 ± 0.10	6.73 ± 0.24
	Search-R1 (original)	0.445	0.516	11.0 ± 0.25	1.50 ± 0.05	2.23 ± 0.14
	DeepResearcher	0.419	0.503	13.3 ± 0.34	7.04 ± 0.33	3.86 ± 0.09
	MEM1-QA (SFT)	0.302	0.358	6.54 ± 0.05	3.30 ± 0.13	4.84 ± 0.21
	MEM1-QA	0.405	0.471	5.63 ± 0.03	0.76 ± 0.02	3.79 ± 0.07
Online Web-QA	Qwen2.5-7B-Inst	0.334	0.451	8.37 ± 0.18	1.39 ± 0.06	2.20 ± 0.04
	DeepResearcher	0.372	0.492	10.27 ± 0.19	2.86 ± 0.14	2.87 ± 0.06
	MEM1-QA	0.397	0.485	5.79 ± 0.06	0.44 ± 0.02	1.84 ± 0.03

4.4. Analysis on Emergent Agent Behaviors and Failure Cases

图3：MEM1 在 2-objective QA 任务中的 emergent behaviors 的 internal states 和 actions 片段。浅蓝色表示与 multi-objective tasks 相关的行为；米色表示与 internal state 中 memory 相关的行为；浅绿色表示与 general search strategies 相关的行为。

Through analyzing MEM1's multi-turn interaction traces trained on 2-objective QA, we observe a range of emergent behaviors that are critical for handling long-horizon, multi-objective tasks, demonstrating capabilities well beyond simple retrieval. Additionally, we provide detailed qualitative and quantitative analysis of cases where MEM1 fails in Appendix G. First, MEM1 learns to manage multiple questions concurrently by maintaining a structured internal state. As shown in Figure 3(a), when faced with two multi-turn questions, the agent stores and updates memory for each question separately, guiding subsequent searches based on the identified information gaps. In (b), MEM1 exhibits the ability to shift focus when progress on one question stalls, recognizing difficulty and prioritizing the more tractable objective. Meanwhile, MEM1 learns to interleave reasoning and memory in its internal state $S$'s, weaving important information into its decision-making process to support both information retention and action selection. In Figure 3(c), MEM1 explicitly extracts important information from previous search results and leverages it to formulate the next query that best addresses the current information gap. In addition, (d) shows that when new, relevant information is retrieved, MEM1 explicitly reasons about its significance and selectively updates its memory. We believe that learning these interleaved behaviors is key to achieving efficiency gains in memory without degrading performance.

通过分析在 2-objective QA 上训练得到的 MEM1 多轮交互轨迹，我们观察到一系列对处理长程、多目标任务至关重要的涌现行为，显示出远超简单检索的能力。 此外，我们在附录 G 中提供了 MEM1 失败案例的详细定性和定量分析。 首先，MEM1 学会通过维护结构化内部状态来并行管理多个问题。 如图3(a) 所示，面对两个多轮问题时，智能体会分别存储和更新每个问题的记忆，并基于识别出的信息缺口指导后续搜索。 在 (b) 中，当某个问题进展停滞时，MEM1 展现出转移关注点的能力，能够识别困难并优先处理更容易推进的目标。 与此同时，MEM1 学会在内部状态 $S$ 中交错组织推理和记忆，把重要信息编织进决策过程，同时支持信息保留和动作选择。 在图3(c) 中，MEM1 会从先前搜索结果中显式提取重要信息，并利用它来形成最能解决当前信息缺口的下一条查询。 此外，(d) 显示，当检索到新的相关信息时，MEM1 会显式推理其重要性，并选择性更新记忆。 我们认为，学习这些交错行为是实现记忆效率提升且不降低性能的关键。

Beyond behaviors unique to our multi-objective setup and memory architecture, MEM1 also exhibits several general-purpose search strategies. In (e), the agent performs self-verification, correcting an earlier misconception and issuing a new query for confirmation. In (f), complex queries are decomposed into manageable subgoals before initiating the search. In (g), for questions requiring multi-turn information gathering, MEM1 extracts key information from search results and uses it to inform the next search. In (h), when overly specific queries fail, MEM1 re-scopes its query to improve retrieval. Notably, many of these behaviors, including verification, making a plan, and iterative search, are also reported in recent studies on deep research agents.

除了我们多目标设置和记忆架构特有的行为外，MEM1 还表现出若干通用搜索策略。 在 (e) 中，智能体会进行自我验证，纠正先前误解并发出新的查询进行确认。 在 (f) 中，复杂查询会在开始搜索前被分解为可管理的子目标。 在 (g) 中，对于需要多轮信息收集的问题，MEM1 会从搜索结果中提取关键信息，并用其指导下一次搜索。 在 (h) 中，当过于具体的查询失败时，MEM1 会重新界定查询范围以改善检索。 值得注意的是，这些行为中的许多，包括验证、制定计划和迭代搜索，也出现在近期 deep research 智能体研究中。

5. Conclusion, Limitations, and Future Work

We introduced MEM1, a reinforcement learning framework that enables language agents to perform long-horizon reasoning with consolidated memory. By integrating inference-time reasoning and memory consolidation into a unified internal state, MEM1 addresses the scalability challenges of prompt growth and achieves competitive performance across QA and web navigation benchmarks, with substantially reduced memory usage and inference latency. Despite these advantages, MEM1 assumes access to environments with well-defined and verifiable rewards. However, many open-ended tasks present ambiguous or noisy reward structures. Fully realizing the potential of MEM1 therefore requires advances in modeling such tasks and designing suitable reward mechanisms--challenges that lie beyond the scope of this work. A promising future direction is to explore methods for training MEM1 agents in open-ended settings where reward signals are sparse, delayed, or implicit.

我们提出了 MEM1，这是一个强化学习框架，使语言智能体能够借助整合记忆执行长程推理。 通过把推理时 reasoning 和记忆整合统一到同一个内部状态中，MEM1 解决了提示增长带来的可扩展性挑战，并在 QA 和网页导航 benchmark 上取得有竞争力的性能，同时显著降低记忆使用和推理延迟。 尽管有这些优势，MEM1 假设环境能够提供明确定义且可验证的奖励。 然而，许多开放式任务具有模糊或有噪声的奖励结构。 因此，要充分释放 MEM1 的潜力，还需要进一步研究如何建模这类任务并设计合适的奖励机制，而这些挑战超出了本文范围。 一个有前景的未来方向，是探索如何在奖励信号稀疏、延迟或隐式的开放式设置中训练 MEM1 agents。