---

Mem-π: Adaptive Memory through Learning When and What to Generate

MemoryAgentRLServiceNowMila蒙特利尔大学蒙特利尔理工学院麦吉尔大学

Wang Xiaoqiang, Wang Chao, Nekoei Hadi, et al. Mem-π: adaptive memory through learning when and what to generate[PP/OL]. V1. arXiv (2026-05-20)[2026-10-23]. DOI:10.48550/arXiv.2605.21463.

https://arxiv.org/abs/2605.21463v1

---

Mem-π：通过学习何时生成及生成什么来实现自适应记忆

Abstract

We present Mem-π, a framework for adaptive memory in large language model (LLM) agents, where useful guidance is generated on demand rather than retrieved from external memory stores. Existing memory-augmented agents typically rely on similarity-based retrieval from episodic memory banks or skill libraries, returning static entries that often misalign with the current query context. In contrast, we model memory as a generative policy realized by a dedicated language or vision-language model with its own parameters, separate from the downstream agent, and fine-tuned specifically to produce context-specific guidance that cues the agent on how to perform complex tasks. The memory policy jointly decides when to produce guidance and what guidance to produce, trained with a decision-content decoupled reinforcement learning (RL) objective so that it abstains when generation would not help and otherwise produces concise, task-relevant guidance. Across diverse agentic benchmarks spanning web navigation, terminal tool use, and embodied environments, Mem-π consistently outperforms retrieval-based and RL-optimized memory baselines, achieving over 20% relative improvement on average.

我们提出 Mem-π，这是一个用于大语言模型（LLM）智能体的自适应记忆框架，其中有用的指导信息是 按需生成 的，而非从外部记忆存储中检索得到。 现有的记忆增强智能体通常依赖基于相似度的检索，从情景记忆库或技能库中返回静态条目，而这些条目往往与当前查询上下文不匹配。 相比之下，我们将记忆建模为生成式策略，由一个拥有独立参数、与下游智能体分离的专用语言模型或视觉语言模型实现，并针对性地微调，使其生成符合具体上下文的指导，提示智能体如何完成复杂任务。 记忆策略联合决定 何时 提供指导以及 提供什么 指导，并采用决策与内容解耦的强化学习（RL）目标进行训练，使其在生成无法带来帮助时选择弃权，否则生成简洁且与任务相关的指导。 在涵盖网页导航、终端工具使用和具身环境的多种智能体基准上，Mem-π 持续优于基于检索和经 RL 优化的记忆基线，平均相对提升超过 20%。

1. Introduction

Large language models (LLMs) have demonstrated remarkable capabilities on reasoning-intensive benchmarks and shown potential as autonomous agents operating in real-world environments, enabling applications such as computer-use agents, deep research assistants, and automated scientific discovery systems. Despite these advances, current LLMs remain limited by their stateless nature and cannot accumulate reusable experience across interactions. To address this limitation, recent agent systems augment LLMs with external memory modules, such as episodic memory banks and reusable skill libraries distilled from prior interactions (Figure 1).

大语言模型（LLM）已在推理密集型基准上展现出卓越能力，并表现出作为自主智能体在真实环境中运行的潜力，推动了计算机使用智能体、深度研究助手和自动化科学发现系统等应用。 尽管取得了这些进展，当前 LLM 仍受无状态特性的限制，无法在多次交互之间积累可复用的经验。 为解决这一局限，近期的智能体系统通过外部记忆模块增强 LLM，例如从既往交互中蒸馏得到的情景记忆库和可复用技能库（图1）。

Existing memory-augmented agents collect memory fragments into a bank and retrieve relevant entries at inference time. Early approaches use workflow-based memory, where predefined rules govern memory construction, retrieval, and update. Recent work explores learning-based memory, optimizing memory operations end-to-end via downstream task outcomes. However, both lines remain constrained by a retrieval-based paradigm that reuses explicitly stored experiences. Retrieved memories are inherently static and often contain irrelevant, partially aligned, or overly specific information that cannot adapt to the agent's current context.

现有记忆增强智能体将记忆片段收集到记忆库中，并在推理时检索相关条目。 早期方法采用 基于工作流的记忆，由预定义规则管理记忆的构建、检索与更新。 近期研究探索 基于学习的记忆，利用下游任务结果端到端地优化记忆操作。 然而，这两条路线都受限于复用显式存储经验的检索范式。 检索得到的记忆本质上是静态的，往往包含无关、仅部分匹配或过于具体的信息，无法适应智能体当前的上下文。

Cognitive science suggests a different view: human remembering is not a literal replay mechanism but a constructive process, where recollection is dynamically reconstructed from prior knowledge and the current context. Concurrent work such as ParaMem and SEAM replaces retrieved memory with generated memory, but either applies it without conditioning on the current context or invokes generation as an always-on auxiliary step. This raises a reliability concern that is especially acute in agent settings, where memory is not the final task output but an intervention on a downstream agent. Under ambiguous, weakly grounded, or out-of-distribution contexts, generated guidance can be uninformative or even harmful, propagating hallucinated cues into agent actions.

认知科学提出了不同的观点：人类的记忆并非逐字回放机制，而是一个 建构性 过程，即根据先前知识和当前上下文动态重建回忆。 ParaMem 和 SEAM 等同期工作用生成式记忆替代检索式记忆，但它们要么不以当前上下文为条件，要么始终将生成作为一个辅助步骤调用。 这带来了可靠性方面的担忧，在智能体场景中尤为突出，因为记忆不是最终的任务输出，而是对下游智能体的一种干预。 面对模糊、依据不足或分布外的上下文时，生成的指导可能毫无信息价值，甚至有害，并将幻觉线索传递到智能体的动作中。

Building on this, we present Mem-π, a framework for adaptive memory generation in LLM agents. Rather than retrieving fixed entries or unconditionally generating auxiliary guidance, Mem-π models memory as a parametric policy ${\pi }_{\mathrm{mem}}$ that learns both when to generate and what to generate. Conditioned on the agent context (i.e., task instructions and environment observations), Mem-π produces concise, task-adaptive guidance from reusable experience encoded in its parameters.

基于这一认识，我们提出 Mem-π，一个面向 LLM 智能体的自适应记忆生成框架。 Mem-π 不检索固定条目，也不无条件生成辅助指导，而是将记忆建模为参数化策略 ${\pi }_{\mathrm{mem}}$，同时学习 何时生成 与 生成什么。 以智能体上下文（即任务指令和环境观测）为条件，Mem-π 从编码在参数中的可复用经验中生成简洁、适应当前任务的指导。

Encoding experience in an Mem-π model's parameters brings several advantages. First, its memory footprint is bounded by model size rather than the number of accumulated experiences, reducing the growing memory-management overhead associated with merging and forgetting. Second, since ${\pi }_{\mathrm{mem}}$ synthesizes guidance on demand rather than copying stored entries, it can fuse signals from many past experiences into a single context-specific hint, unlike top-k retrieval, which may split them across fragments or omit them beyond the cutoff. Finally, this framework separates specialization from execution: a smaller private local model can be fine-tuned as ${\pi }_{\mathrm{mem}}$ and plugged into a larger or frontier agent model to leverage broader reasoning capabilities.

将经验编码到 Mem-π 模型的参数中具有多项优势。 首先，其记忆占用受模型规模约束，而非受累积经验数量影响，从而减少了记忆合并与遗忘所带来的、不断增长的管理开销。 其次，由于 ${\pi }_{\mathrm{mem}}$ 按需综合生成指导，而不是复制已存储的条目，它可以将多段既往经验中的信号融合为一条符合当前上下文的提示；相比之下，top-k 检索可能将这些信号分散在多个片段中，或因截断而遗漏它们。 最后，该框架将专门化与执行分离：可以将一个较小的私有本地模型微调为 ${\pi }_{\mathrm{mem}}$，再接入更大或前沿的智能体模型，利用后者更广泛的推理能力。

We train Mem-π in two stages. Experience distillation first compresses an offline experience bank into the memory policy via supervised learning, internalizing reusable behaviors. Adaptation distillation then refines the policy through reinforcement learning, using downstream task outcomes as the reward signal to align memory generation with task success. To ensure reliability, we incorporate abstention into ${\pi }_{\mathrm{mem}}$, allowing it to skip memory generation when generation is unnecessary or uncertain. Specifically, we introduce a decision-content decoupled objective built on Group Relative Policy Optimization (GRPO) that separates when to generate from what to generate. The objective uses structured counterfactual rollouts to compare the two branches, decomposing learning into decision-level and content-level advantages and enabling adaptive memory behavior: the policy generates guidance only when it improves downstream task outcomes, and abstains otherwise.

我们分两个阶段训练 Mem-π。 经验蒸馏 首先通过监督学习将离线经验库压缩到记忆策略中，内化可复用的行为。 随后，适应蒸馏 通过强化学习进一步优化策略，以下游任务结果作为奖励信号，使记忆生成与任务成功相一致。 为确保可靠性，我们在 ${\pi }_{\mathrm{mem}}$ 中引入 弃权 机制，允许策略在生成没有必要或存在不确定性时跳过记忆生成。 具体而言，我们提出基于组相对策略优化（GRPO）的决策与内容解耦目标，将 何时生成 与 生成什么 分离。 该目标利用结构化反事实轨迹采样比较两个分支，将学习信号分解为决策级和内容级优势，从而实现自适应的记忆行为：策略仅在指导能够改善下游任务结果时生成，否则选择弃权。

We evaluate Mem-π across diverse agent benchmarks spanning web navigation (WebArena, WorkArena), terminal tool use (LifelongAgentBench), and text-based embodied environments (ALFWorld). Adaptive memory generation consistently outperforms retrieval-based memory baselines across all four benchmarks, yielding a 20% relative gain over the base agent on average, with the relative gain on WebArena approaching 50%.

我们在多种智能体基准上评估 Mem-π，涵盖网页导航（WebArena、WorkArena）、终端工具使用（LifelongAgentBench）和基于文本的具身环境（ALFWorld）。 自适应记忆生成在全部四个基准上持续优于基于检索的记忆基线，相比基础智能体平均取得 20% 的相对提升，其中 WebArena 上的相对提升接近 50%。

图1：三种记忆系统的比较：（a）基于工作流的记忆系统，其记忆操作由预定义的检索与更新流程管理；（b）基于学习的记忆系统，其记忆操作与下游智能体结果联合优化；（c）我们提出的 Mem-π，将记忆建模为独立于下游智能体的生成式策略 πₘₑₘ，并通过离线经验蒸馏和在线适应蒸馏内化可复用经验。

图2：Mem-π 概览。我们分两个阶段训练生成式记忆策略 πₘₑₘ。经验蒸馏通过监督学习从离线经验库中提炼可复用经验。随后，适应蒸馏依据下游智能体结果优化 πₘₑₘ，采用决策与内容解耦的策略优化，将弃权与生成的轨迹采样配对，并把 GRPO 优势分解为决策级和内容级信号。

2. Design of Mem-π

We model adaptive memory as a generative policy ${\pi }_{\mathrm{mem}}$ parameterized by $\theta$ and instantiated as a dedicated language or vision-language model Mem-π, separate from the downstream agent. Mem-π produces guidance that is injected into the agent's context at inference time. Let $\mathcal{E}$ denote an offline experience bank of context-guidance pairs $(x,m)$, where each task context $x=(q,o)$ consists of a task specification $q$ and an environment observation $o\in \mathcal{O}$, and each memory guidance $m\in \mathcal{M}$ is a textual hint inserted into the downstream agent's context to inform its decisions. The observation $o$ may include structured textual representations and, when available, visual inputs such as screenshots in web navigation tasks. Figure 8 illustrates the structure of each field.

我们将自适应记忆建模为由 $\theta$ 参数化的生成式策略 ${\pi }_{\mathrm{mem}}$，并将其实现为与下游智能体分离的专用语言模型或视觉语言模型 Mem-π。 Mem-π 生成指导信息，并在推理时将其注入智能体上下文。 令 $\mathcal{E}$ 表示由上下文—指导对 $(x,m)$ 构成的离线经验库，其中每个任务上下文 $x=(q,o)$ 包含任务描述 $q$ 和环境观测 $o\in \mathcal{O}$，而每条记忆指导 $m\in \mathcal{M}$ 都是一段插入下游智能体上下文、为其决策提供依据的文本提示。 观测 $o$ 可以包含结构化文本表示，也可以在可用时包含视觉输入，例如网页导航任务中的截图。 图8 展示了各字段的结构。

First, experience distillation learns a mapping ${\pi }_{\mathrm{mem}}^1:(q,o)\mapsto m$ via supervised learning on $\mathcal{E}$, converting explicit offline experiences into parametric knowledge so that the policy can produce context-specific guidance for new tasks at inference time. This design is inspired by context-supervised pretraining, where models learn to reconstruct knowledge from context and internalize it into their parameters. Let $m_t$ denote the $t$-th token of the target memory $m$, and let $m_{<t}$ denote its preceding tokens. We optimize ${\pi }_{\mathrm{mem}}^1$ with the autoregressive supervised objective:

首先，经验蒸馏 在 $\mathcal{E}$ 上通过监督学习获得映射 ${\pi }_{\mathrm{mem}}^1:(q,o)\mapsto m$，将显式的离线经验转化为参数化知识，使策略能够在推理时为新任务生成符合具体上下文的指导。 这一设计受到上下文监督预训练的启发：模型学习根据上下文重建知识，并将其内化到参数中。 令 $m_t$ 表示目标记忆 $m$ 的第 $t$ 个 token，$m_{<t}$ 表示它之前的 token。 我们使用以下自回归监督目标优化 ${\pi }_{\mathrm{mem}}^1$：

$$\begin{array}{}\text{(1)}& {\mathcal{J}}_{\mathrm{mem}}^{(1)}(\theta )={\mathbb{E}}_{(x,m)\sim \mathcal{E}}[\sum _{t=1}^{|m|}\log {\pi }_{\mathrm{mem}}^1(m_t\mid x,m_{<t};\theta )].\end{array}$$

Second, adaptation distillation (Section 2.1) initializes ${\pi }_{\mathrm{mem}}^2$ from ${\pi }_{\mathrm{mem}}^1$ and further optimizes the shared parameters $\theta$ through reinforcement learning from downstream agent outcomes, aligning memory generation with task utility rather than imitation quality alone. To support reliable memory use, we introduce an explicit abstention decision, enabling the policy to skip generation when guidance is unnecessary or potentially unhelpful. Specifically, we extend the output space with a decision token and define the mapping ${\pi }_{\mathrm{mem}}^2:(q,o)\mapsto y$, where $y=d\oplus m$, $d\in \{\mathtt{[}\mathtt{GENERATE}\mathtt{]},\mathtt{[}\mathtt{ABSTAIN}\mathtt{]}\}$, $m\in \mathcal{M}\cup \{\varnothing \}$, and $\oplus$ denotes string concatenation. When $d=\mathtt{[}\mathtt{GENERATE}\mathtt{]}$, the policy emits guidance $m\in \mathcal{M}$, which is prepended to the downstream agent input to form the augmented context $x\oplus m$. When $d=\mathtt{[}\mathtt{ABSTAIN}\mathtt{]}$, we set $m=\varnothing$, and the agent operates on the original context $x$.

其次，适应蒸馏（第 2.1 节）以 ${\pi }_{\mathrm{mem}}^1$ 初始化 ${\pi }_{\mathrm{mem}}^2$，并根据下游智能体结果，通过强化学习进一步优化共享参数 $\theta$，使记忆生成与任务效用相一致，而非仅关注模仿质量。 为支持可靠的记忆使用，我们引入显式的弃权决策，使策略在指导没有必要或可能无益时跳过生成。 具体而言，我们通过一个决策 token 扩展输出空间，并定义映射 ${\pi }_{\mathrm{mem}}^2:(q,o)\mapsto y$，其中 $y=d\oplus m$、$d\in \{\mathtt{[}\mathtt{GENERATE}\mathtt{]},\mathtt{[}\mathtt{ABSTAIN}\mathtt{]}\}$、$m\in \mathcal{M}\cup \{\varnothing \}$，$\oplus$ 表示字符串拼接。 当 $d=\mathtt{[}\mathtt{GENERATE}\mathtt{]}$ 时，策略输出指导 $m\in \mathcal{M}$，将其置于下游智能体输入之前，形成增强上下文 $x\oplus m$。 当 $d=\mathtt{[}\mathtt{ABSTAIN}\mathtt{]}$ 时，我们令 $m=\varnothing$，智能体使用原始上下文 $x$ 运行。

A key challenge in Stage 2 is the imbalance between the routing decision and the memory content. The decision is encoded by a short token prefix, whereas the generated guidance spans a much longer sequence. As a result, under a flat policy-gradient objective, content-level gradients can dominate decision-level learning. We introduce a decision-content decoupled objective (Section 2.2) that separates routing and content learning signals through decomposing flat advantage.

阶段 2 的一个关键挑战在于路由决策与记忆内容之间的不平衡。 决策由较短的 token 前缀编码，而生成的指导则是长得多的序列。 因此，在统一的策略梯度目标下，内容级梯度可能主导决策级学习。 我们引入 决策与内容解耦 的目标（第 2.2 节），通过分解统一优势，分离路由与内容的学习信号。

2.1. Adaptation Distillation

While experience distillation provides a strong initialization, the supervised policy cannot determine when memory generation is useful or potentially harmful. Moreover, its guidance remains bounded by the offline experience bank and is not directly optimized for the needs of the downstream agent. The adaptation distillation addresses this by refining ${\pi }_{\text{mem}}$ with RL using agent outcomes as the reward signal. We extend the model vocabulary with two special tokens, i.e., decision tokens including [GENERATE] and [ABSTAIN], and initialize their embeddings symmetrically so that both decisions have comparable initial probabilities and can be sufficiently explored at the beginning of training.

尽管经验蒸馏提供了良好的初始化，经过监督学习的策略却无法判断记忆生成 何时 有用，或何时可能有害。 此外，其指导仍受离线经验库限制，并未直接针对下游智能体的需求进行优化。 适应蒸馏通过 RL 进一步优化 ${\pi }_{\text{mem}}$，以智能体结果作为奖励信号，从而解决这一问题。 我们在模型词表中加入两个特殊 token，即决策 token [GENERATE] 和 [ABSTAIN]，并对称地初始化它们的嵌入，使两种决策具有相近的初始概率，从而在训练初期得到充分探索。

We adopt GRPO as the base RL algorithm, which removes the need for value models by estimating advantages from grouped samples. For each $x$, GRPO samples a group of $G$ outputs $\{y^1,\dots ,y^G\}$ from ${\pi }_{\text{mem}}$ and computes group-relative advantages ${\hat{A}}^j={\displaystyle \frac{r^j-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})+{ϵ}_{\mathrm{std}}}}$, where $\mathbf{r}=(r^1,\dots ,r^G)$. The policy is updated by maximizing:

我们采用 GRPO 作为基础 RL 算法，它通过成组样本估计优势，无需价值模型。 对于每个 $x$，GRPO 从 ${\pi }_{\text{mem}}$ 中采样一组包含 $G$ 个输出的样本 $\{y^1,\dots ,y^G\}$，并计算组相对优势 ${\hat{A}}^j={\displaystyle \frac{r^j-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})+{ϵ}_{\mathrm{std}}}}$，其中 $\mathbf{r}=(r^1,\dots ,r^G)$。 通过最大化以下目标来更新策略：

$$\begin{array}{}\text{(2)}& \begin{array}{rl}{\mathcal{J}}_{\mathrm{GRPO}}(\theta )& ={\mathbb{E}}_x[\frac{1}{G}\sum _{j=1}^G\frac{1}{|y^j|}\sum _{t=1}^{|y^j|}\\ & (min({\rho }_t^j{\hat{A}}^j,\\ & \mathrm{clip}({\rho }_t^j,1-{ϵ}_{\mathrm{clip}},1+{ϵ}_{\mathrm{clip}}){\hat{A}}^j)\\ & -\beta D_{\mathrm{KL}}^{(t)})],\end{array}\end{array}$$

where ${\rho }_t^j={\displaystyle \frac{{\pi }_{\text{mem}}^2(y_t^j\mid x,y_{<t}^j;\theta )}{{\pi }_{\text{old}}(y_t^j\mid x,y_{<t}^j;{\theta }_{\text{old}})}}$ is the importance ratio between the current policy and the old policy used for rollout sampling, and $D_{\mathrm{KL}}^{(t)}$ denotes the token-level KL divergence from a reference policy ${\pi }_{\text{ref}}$. Here, ${\pi }_{\text{old}}$ is a frozen snapshot of ${\pi }_{\mathrm{mem}}^2$, and ${\pi }_{\mathrm{ref}}$ is set to the Stage-1 policy ${\pi }_{\mathrm{mem}}^1$ before adaptation distillation.

其中，${\rho }_t^j={\displaystyle \frac{{\pi }_{\text{mem}}^2(y_t^j\mid x,y_{<t}^j;\theta )}{{\pi }_{\text{old}}(y_t^j\mid x,y_{<t}^j;{\theta }_{\text{old}})}}$ 是当前策略与用于轨迹采样的旧策略之间的重要性比率，$D_{\mathrm{KL}}^{(t)}$ 表示相对于参考策略 ${\pi }_{\text{ref}}$ 的 token 级 KL 散度。 这里，${\pi }_{\text{old}}$ 是 ${\pi }_{\mathrm{mem}}^2$ 的冻结快照，${\pi }_{\mathrm{ref}}$ 则在适应蒸馏开始前设为阶段 1 策略 ${\pi }_{\mathrm{mem}}^1$。

Reward design. The reward $r=R(x,y)$ consists of a downstream task reward and, for generated memories, a length regularizer $R_m$. Given $y=d\oplus m$, we define:

奖励设计。 奖励 $r=R(x,y)$ 包含下游任务奖励；对于生成的记忆，还包含长度正则项 $R_m$。 给定 $y=d\oplus m$，我们定义：

$$\begin{array}{}\text{(3)}& R(x,y)=\{\begin{array}{l}\begin{array}{rl}& \mathrm{TaskReward}({\pi }_{\mathrm{agent}}(\cdot \mid x\oplus m))+R_m(m),\\ & \text{if }d=\mathtt{[}\mathtt{GENERATE}\mathtt{]}\end{array}\\ \begin{array}{rl}& \mathrm{TaskReward}({\pi }_{\mathrm{agent}}(\cdot \mid x)),\\ & \text{if }d=\mathtt{[}\mathtt{ABSTAIN}\mathtt{]},\end{array}\end{array}\end{array}$$

where ${\pi }_{\mathrm{agent}}$ denotes the downstream agent's action distribution, which is not trained in this stage, and $\mathrm{TaskReward}(\cdot )\in \{0,1\}$ is a binary signal indicating task success or failure from the agent's interaction trajectory under the memory-augmented or original context. Following length-aware reward shaping in reasoning and agentic LLMs, we use $R_m(m)=-{\lambda }_{\text{len}}|m|/L_{max}$ to discourage verbose or overly specific guidance, where $|m|$ is the number of memory tokens, $L_{max}$ is the generation budget, and ${\lambda }_{\text{len}}>0$ controls the penalty.

其中，${\pi }_{\mathrm{agent}}$ 表示下游智能体的动作分布，该阶段不训练此分布；$\mathrm{TaskReward}(\cdot )\in \{0,1\}$ 是二元信号，根据智能体在记忆增强上下文或原始上下文中的交互轨迹，表示任务成功或失败。 遵循推理型和智能体 LLM 中的长度感知奖励塑形方法，我们采用 $R_m(m)=-{\lambda }_{\text{len}}|m|/L_{max}$ 来抑制冗长或过于具体的指导，其中 $|m|$ 是记忆 token 数量，$L_{max}$ 是生成预算，${\lambda }_{\text{len}}>0$ 控制惩罚强度。

2.2. Decision-Content Decoupled Policy Optimization

Applying standard GRPO directly to the structured output $y=d\oplus m$ conflates two distinct learning signals: $d$ governs whether memory is generated, while $m$ governs what guidance is produced. This conflation creates two challenges. First, since Stage 2 is initialized from a supervised policy that favors generation, standard i.i.d. sampling may yield groups with no abstain rollouts, eliminating any comparison between generation and abstention. Second, the length imbalance between $d$ and $m$ causes content-level updates to dominate the flat per-token objective, suppressing the decision-token gradient. To address both, we propose decision-content decoupled policy optimization, which uses structured counterfactual rollouts to decompose the GRPO advantage into decision- and content-level signals and routes each to the corresponding token positions.

直接对结构化输出 $y=d\oplus m$ 应用标准 GRPO，会混合两种不同的学习信号：$d$ 控制 是否 生成记忆，而 $m$ 控制生成 什么 指导。 这种混合带来两项挑战。 首先，由于阶段 2 由偏向生成的监督策略初始化，标准的 独立同分布 采样可能产生完全不包含弃权轨迹的样本组，从而无法比较生成与弃权。 其次，$d$ 与 $m$ 的长度不平衡使内容级更新主导统一的逐 token 目标，抑制决策 token 的梯度。 为解决这两个问题，我们提出 决策与内容解耦的策略优化，利用结构化反事实轨迹采样，将 GRPO 优势分解为决策级和内容级信号，并将各自信号传递到对应的 token 位置。

Structured counterfactual rollout. For each context $x$, we construct a structured rollout group with one abstain branch and $G-1$ generate branches:

结构化反事实轨迹采样。 对于每个上下文 $x$，我们构造一个结构化采样组，其中包含一个弃权分支和 $G-1$ 个生成分支：

$$\begin{array}{}\text{(4)}& \begin{array}{rl}{y}^0& =\mathtt{[}\mathtt{ABSTAIN}\mathtt{]}\oplus \varnothing ,\\ y^j& =\mathtt{[}\mathtt{GENERATE}\mathtt{]}\oplus m^j,j=1,\dots ,G-1.\end{array}\end{array}$$

This guarantees that each group contains both decisions, making the relative value of memory generation versus abstention directly observable. Since abstention has no guidance content to sample, a single abstain rollout suffices, while the remaining rollouts explore diverse generated memories.

这保证了每个组都包含两种决策，使记忆生成与弃权的相对价值能够被直接观测。 由于弃权没有需要采样的指导内容，一个弃权轨迹就已足够，其余轨迹则用于探索多样化的生成记忆。

Decision-content advantage decomposition. Given the structured rollout group, we decompose the learning signal into a cross-branch decision advantage and a within-branch content advantage:

决策与内容优势分解。 给定结构化采样组，我们将学习信号分解为跨分支的决策优势和分支内的内容优势：

$$\begin{array}{}\text{(5)}& \begin{array}{rl}{V}_{\mathrm{abs}}& =R(x,y^0),\\ V_{\mathrm{gen}}& =\frac{1}{G-1}\sum _{j=1}^{G-1}R(x,y^j),\\ \mathrm{\Delta }& =V_{\mathrm{abs}}-V_{\mathrm{gen}}.\end{array}\end{array}$$

Here, $\mathrm{\Delta }$ captures the relative benefit of abstaining over generating memory for the current context. The decision advantage $A_d^j$ uses $\mathrm{\Delta }$ as a signed cross-branch signal: $A_d^j=+\mathrm{\Delta }$ for the abstain rollout ($j=0$) and $A_d^j=-\mathrm{\Delta }$ for generate rollouts ($j\ge 1$). Since $\mathrm{\Delta }=V_{\mathrm{abs}}-V_{\mathrm{gen}}$, abstention receives positive advantage when it outperforms generation, and generation is favored otherwise.

这里，$\mathrm{\Delta }$ 刻画在当前上下文中，弃权相对于生成记忆的收益。 决策 优势 $A_d^j$ 将 $\mathrm{\Delta }$ 用作带符号的跨分支信号：对于弃权轨迹（$j=0$），$A_d^j=+\mathrm{\Delta }$；对于生成轨迹（$j\ge 1$），$A_d^j=-\mathrm{\Delta }$。 由于 $\mathrm{\Delta }=V_{\mathrm{abs}}-V_{\mathrm{gen}}$，当弃权优于生成时，弃权获得正优势；反之，则倾向于生成。

The content advantage $A_c^j$ ranks generated memories via group normalization within the generate branch: $A_c^j=0$ for $j=0$, and $A_c^j={\displaystyle \frac{R(x,y^j)-V_{\mathrm{gen}}}{\mathrm{std}({\mathbf{r}}_{\mathrm{gen}})+{ϵ}_{\mathrm{std}}}}$ for $j\ge 1$, where ${\mathbf{r}}_{\mathrm{gen}}=\{R(x,y^j){\}}_{j=1}^{G-1}$ denotes the rewards of generate rollouts. That is, this term reduces to the standard GRPO formulation within the generate rollouts.

内容 优势 $A_c^j$ 通过生成分支内的组归一化，对生成的记忆进行排序：当 $j=0$ 时，$A_c^j=0$；当 $j\ge 1$ 时，$A_c^j={\displaystyle \frac{R(x,y^j)-V_{\mathrm{gen}}}{\mathrm{std}({\mathbf{r}}_{\mathrm{gen}})+{ϵ}_{\mathrm{std}}}}$，其中 ${\mathbf{r}}_{\mathrm{gen}}=\{R(x,y^j){\}}_{j=1}^{G-1}$ 表示生成轨迹的奖励。 也就是说，这一项在生成轨迹内退化为标准 GRPO 形式。

Token-level credit assignment. To route the decomposed signals to the appropriate token positions, we construct a per-token advantage $A_t^j$. Let $T_d$ denote the length of the decision prefix. We assign

Token 级信用分配。 为将分解后的信号传递到恰当的 token 位置，我们构造逐 token 优势 $A_t^j$。 令 $T_d$ 表示决策前缀的长度。 我们赋值为

$$\begin{array}{}\text{(6)}& A_t^j=\{\begin{array}{ll}{A}_d^j,& t\le T_d\\ \mathbf{1}[\mathrm{\Delta }<0]A_c^j,& t>T_d\end{array}\end{array}$$

Decision tokens receive only the decision-level signal $A_d^j$, while content tokens receive the content-level signal $A_c^j$ only when generation improves over abstention, i.e., $\mathrm{\Delta }<0$. This $\mathrm{\Delta }$-gating avoids updating generated content in contexts where memory generation is not beneficial, preventing the assignment of content-level gradients to suboptimal generate decisions. Substituting $A_t^j$ into the GRPO objective yields the Stage 2 adaptation objective:

决策 token 仅接收决策级信号 $A_d^j$，而内容 token 只有在生成优于弃权时，即 $\mathrm{\Delta }<0$ 时，才接收内容级信号 $A_c^j$。 这一 $\mathrm{\Delta }$ 门控机制避免在记忆生成无益的上下文中更新生成内容，从而防止为次优的生成决策分配内容级梯度。 将 $A_t^j$ 代入 GRPO 目标，得到阶段 2 的适应目标：

$$\begin{array}{}\text{(7)}& \begin{array}{rl}{\mathcal{J}}_{\mathrm{mem}}^{(2)}(\theta )& ={\mathbb{E}}_x[\frac{1}{G}\sum _{j=0}^{G-1}\frac{1}{|y^j|}\sum _{t=1}^{|y^j|}\\ & (min({\rho }_t^j{A}_t^j,\\ & \mathrm{clip}({\rho }_t^j,1-{ϵ}_{\mathrm{clip}},1+{ϵ}_{\mathrm{clip}})A_t^j)\\ & -\beta D_{\mathrm{KL}}^{(t)})].\end{array}\end{array}$$

Compared with standard GRPO (Eq. 2), the only objective-level change is replacing the scalar group-relative advantage ${\hat{A}}^j$ with the per-token advantage $A_t^j$. This preserves the GRPO framework while separating the two learning problems: decision tokens learn when to generate through cross-branch comparison, and content tokens learn what to generate through within-branch ranking.

与标准 GRPO（式（2））相比，目标层面唯一的变化是用逐 token 优势 $A_t^j$ 替换标量组相对优势 ${\hat{A}}^j$。 这保留了 GRPO 框架，同时将两项学习问题分离：决策 token 通过跨分支比较学习 何时 生成，内容 token 通过分支内排序学习生成 什么。

3. Experiments

Benchmarks. We evaluate on four agentic benchmarks. WebArena contains 812 multi-step browser tasks over five domains (Shopping, CMS, GitLab, Reddit, Maps). Following WebAgent-R1 and WebRL, we use a 647/165 train/test split. WorkArena is an enterprise software web-navigation benchmark on the ServiceNow platform, covering 33 task templates across four categories (Menu, Form, List, Knowledge). We use 20 seeds per template for training and 10 disjoint seeds for evaluation.

基准。 我们在四个智能体基准上进行评估。 WebArena 包含五个领域（Shopping、CMS、GitLab、Reddit、Maps）的 812 个多步浏览器任务。 遵循 WebAgent-R1 和 WebRL，我们将其中 647 个任务用于训练，165 个任务用于测试。 WorkArena 是 ServiceNow 平台上的企业软件网页导航基准，涵盖四类任务（Menu、Form、List、Knowledge），共 33 个任务模板。 我们为每个模板使用 20 个随机种子进行训练，并使用另外 10 个互不重叠的种子进行评估。

LifelongAgentBench (LAB) tests experience reuse in terminal environments. Following MemRL, we use the Database (DB, 22 SQL skills) and Operating System (OS, 29 Bash skills) subsets, each with 500 tasks and a 7:3 split. ALFWorld consists of text-based embodied household tasks across six manipulation types. We follow the official split with 3,553 train and 134 unseen test tasks.

LifelongAgentBench（LAB） 测试终端环境中的经验复用能力。 遵循 MemRL，我们使用数据库（DB，22 项 SQL 技能）和操作系统（OS，29 项 Bash 技能）子集，每个子集包含 500 个任务，并按 7:3 划分。 ALFWorld 包含基于文本的具身家务任务，覆盖六种操作类型。 我们遵循官方划分，使用 3,553 个训练任务和 134 个未见测试任务。

We use task success rate (SR) as the reward signal across all benchmarks. We construct the offline experience bank using JEF-Hinter, which distills raw interaction traces into compact, reusable hints by identifying decisive steps in long trajectories. We emphasize that our Mem-π framework is source-agnostic. Any retrieval-based memory bank, including human demonstrations, agent traces, or curated documentation, can serve as supervision for ${\pi }_{\mathrm{mem}}$, effectively converting retrieval-based memory into a generative one.

所有基准均使用任务成功率（SR）作为奖励信号。 我们利用 JEF-Hinter 构建离线经验库，它通过识别长轨迹中的关键步骤，将原始交互轨迹蒸馏为紧凑、可复用的提示。 我们强调，Mem-π 框架不依赖特定的经验来源。 任何基于检索的记忆库，包括人工演示、智能体轨迹或整理后的文档，都可以作为 ${\pi }_{\mathrm{mem}}$ 的监督信号，从而有效地将检索式记忆转化为生成式记忆。

Baselines. Beyond the base agents (with no memory), we compare against two memory paradigms. (i) Workflow-based memory: RAG retrieves the top-k experiences from the JEF-Hinter memory bank via BM25, effectively matching the approach used in JEF-Hinter. Mem0 combines RAG with rule-based management. In both settings, we fix $k=1$. (ii) Learning-based memory: Memory-R1 trains a memory manager with outcome-driven RL for structured memory operations. MemRL learns Q-values over episodic memory for utility-aware retrieval.

基线。 除了不使用记忆的基础智能体，我们还与两种记忆范式进行比较。 （i）基于工作流的记忆：RAG 通过 BM25 从 JEF-Hinter 记忆库检索 top-k 条经验，实质上与 JEF-Hinter 使用的方法一致。 Mem0 将 RAG 与基于规则的管理相结合。 在这两种设置中，我们都固定 $k=1$。 （ii）基于学习的记忆：Memory-R1 利用结果驱动的 RL 训练记忆管理器，执行结构化记忆操作。 MemRL 在情景记忆上学习 Q 值，实现效用感知的检索。

Agent and memory configuration. The memory model Mem-π and the downstream agent are two separate models with independent parameters, even when they share the same backbone architecture. For a fair comparison with Memory-R1, we adopt the same backbone, Qwen-2.5-7B-Instruct, for the memory model ${\pi }_{\mathrm{mem}}$, on which we apply Mem-π's two-stage distillation. Training-based methods use only training-split tasks and their corresponding JEF-Hinter hints. The same split isolation is applied to the RAG and Mem0 banks.

智能体与记忆配置。 即使采用相同的骨干架构，记忆模型 Mem-π 与下游智能体也是两个参数独立的不同模型。 为与 Memory-R1 公平比较，我们为记忆模型 ${\pi }_{\mathrm{mem}}$ 采用相同的骨干模型 Qwen-2.5-7B-Instruct，并在其上执行 Mem-π 的两阶段蒸馏。 需要训练的方法仅使用训练集任务及其对应的 JEF-Hinter 提示。 RAG 和 Mem0 的记忆库也采用相同的数据集隔离方式。

All results are evaluated on held-out test tasks. To assess cross-agent generalization, we evaluate two downstream agents: (i) a Qwen-2.5-7B-Instruct agent fine-tuned under the same setting as WebAgent-R1, also used during stage-2 adaptation distillation, and (ii) the proprietary gpt-5.4-mini. Section 3 reports text-only results with gpt-5.4-mini as the base agent. Section 3.3 further reports cross-agent evaluation and visual-input ablations on WebArena. In the visual setting, the memory model receives the initial screenshot and visual grounding extracted by gemini-2.5-flash, using Qwen-2.5-VL-7B-Instruct as the visual backbone. Implementation details are in Appendix A.

所有结果均在留出的测试任务上评估。 为评估跨智能体泛化，我们测试两个下游智能体：（i）在与 WebAgent-R1 相同设置下微调的 Qwen-2.5-7B-Instruct 智能体，它也用于阶段 2 的适应蒸馏；（ii）闭源模型 gpt-5.4-mini。 第 3 节 报告以 gpt-5.4-mini 为基础智能体的纯文本结果。 第 3.3 节 进一步报告跨智能体评估以及 WebArena 上的视觉输入消融实验。 在视觉设置中，记忆模型接收初始截图以及由 gemini-2.5-flash 提取的视觉定位信息，并使用 Qwen-2.5-VL-7B-Instruct 作为视觉骨干模型。 实现细节见 附录 A。

3.1. Main Results

表1：以 gpt-5.4-mini 为基础智能体，在四个智能体基准上的任务成功率（SR，%）。粗体表示各列最佳结果，下划线表示次优结果。

Method	WebArena						WorkArena					ALFWorld							LAB			Avg.
	Shop	CMS	GL	Red	Map	Avg	M&D	Form	Filt	SK	Avg	P&P	Exam	Cln	Heat	Cool	P2P	Avg	DB	OS	Avg
Base Agent	28.4	14.6	31.2	28.8	32.4	27.1	31.9	55.7	11.6	76.9	42.0	88.3	82.7	86.1	85.0	85.5	78.8	85.3	28.5	25.0	26.8	45.3
RAG	28.6	20.2	37.4	38.2	32.4	31.4	33.6	60.6	9.0	78.1	42.6	89.1	83.4	86.8	85.5	85.8	79.6	87.1	29.9	27.2	28.5	47.4
Mem0	29.8	22.4	38.6	36.4	32.2	31.9	33.9	61.8	9.5	79.1	44.1	89.7	84.1	87.3	85.8	87.2	80.5	87.5	31.9	28.1	30.0	48.4
Memory-R1	30.6	24.8	40.2	38.8	31.6	33.2	35.4	62.3	10.5	80.5	44.3	89.9	85.0	87.7	86.3	87.7	81.1	87.9	32.6	29.8	31.2	49.2
MemRL	31.2	26.4	41.8	40.0	30.8	34.0	36.5	63.6	10.6	80.5	46.1	90.8	85.5	88.7	86.5	87.1	81.2	88.0	33.0	30.7	31.9	50.0
Mem-π (Stage 1)	30.6	17.4	46.8	47.8	32.4	35.0	41.6	65.8	9.0	81.5	46.6	92.1	88.0	91.2	89.0	90.0	84.1	90.0	35.3	32.9	34.1	51.4
Mem-π	34.6	42.8	50.2	52.6	35.4	43.1	45.1	70.6	13.6	85.3	50.3	94.2	90.2	92.3	91.5	91.6	86.7	91.6	38.4	35.0	36.7	55.4

Mem-π achieves state-of-the-art performance across all benchmarks and sub-domains. As summarized in Table 1, Mem-π leads every WebArena sub-domain, with the largest absolute gains in Reddit (+23.8 pp) and CMS (+28.2 pp), where structured navigation patterns benefit most from memorized experience. On WorkArena, Mem-π improves the base agent from 42.0% to 50.3% on average, with strong gains on Form (+14.9 pp). On ALFWorld, Mem-π achieves 91.6%, a +6.3 pp improvement over the already-strong GPT-5.4-mini baseline.

Mem-π 在所有基准及子领域上均达到当前最优性能。 如 表1 所示，Mem-π 在 WebArena 的所有子领域上均领先，其中 Reddit（+23.8 个百分点）和 CMS（+28.2 个百分点）的绝对提升最大，这些领域的结构化导航模式最能从记忆经验中受益。 在 WorkArena 上，Mem-π 将基础智能体的平均成功率从 42.0% 提升至 50.3%，其中 Form 的提升尤为显著（+14.9 个百分点）。 在 ALFWorld 上，Mem-π 达到 91.6%，相比已经很强的 GPT-5.4-mini 基线提升了 6.3 个百分点。

Experience distillation alone already matches or surpasses RL-based baselines. Mem-π (Stage 1) achieves 35.0% on WebArena, comparable to Memory-R1 (33.2%) and MemRL (34.0%) without any RL training. This validates offline parametric knowledge as a strong initialization strategy.

仅依靠经验蒸馏，就已能够匹敌或超过基于 RL 的基线。 Mem-π（阶段 1）在 WebArena 上达到 35.0%，无需任何 RL 训练便可与 Memory-R1（33.2%）和 MemRL（34.0%）相媲美。 这验证了离线参数化知识是一种有效的初始化策略。

The RL stage provides significant additional gains on WebArena. Moving from Stage 1 to the full model yields +8.1 pp on WebArena overall, with the largest jumps on CMS (+25.4 pp), Reddit (+4.8 pp), and Maps (+3.0 pp). ALFWorld gains a more modest +1.6 pp, consistent with the frontier agent's high baseline leaving less room for improvement.

RL 阶段为 WebArena 带来显著的额外收益。 从阶段 1 模型升级为完整模型后，WebArena 的整体成功率提升了 8.1 个百分点，其中 CMS（+25.4 个百分点）、Reddit（+4.8 个百分点）和 Maps（+3.0 个百分点）的提升最大。 ALFWorld 的提升较小，为 1.6 个百分点，这与前沿智能体较高的基线水平、较小的提升空间相一致。

3.2. Ablation Study

RQ1: Are both training stages necessary? We compare Mem-π against two single-stage variants. (i) w/o Stage 1 init skips the experience distillation (SFT phase) and applies online RL directly to Qwen2.5-7B-Instruct. (ii) Unified single-stage collapses both stages into one RL phase that jointly optimizes the downstream task reward $R_{\text{task}}$, the same length regularizer $R_m$ used in Mem-π, and an additional BERTScore-based similarity reward $R_{\text{sim}}$ computed between the generated memory and the corresponding reference guidance from the training bank, so that the single RL stage has both an imitation signal toward reference hints and a downstream task signal.

RQ1：两个训练阶段都必要吗？ 我们将 Mem-π 与两种单阶段变体进行比较。 （i）移除阶段 1 初始化 跳过经验蒸馏（SFT 阶段），直接对 Qwen2.5-7B-Instruct 进行在线 RL。 （ii）统一的单阶段训练 将两个阶段合并为一个 RL 阶段，联合优化下游任务奖励 $R_{\text{task}}$、与 Mem-π 相同的长度正则项 $R_m$，以及一个额外的、基于 BERTScore 的相似度奖励 $R_{\text{sim}}$；该奖励在生成记忆与训练库中对应的参考指导之间计算，使单一 RL 阶段同时获得模仿参考提示的信号和下游任务信号。

Results in Table 2 show that both training stages are essential, with unified training suffering the largest drop. Removing Stage 1 initialization degrades WebArena by 5.2 pp, suggesting that without a well-initialized memory distribution, online RL struggles to converge. Unified single-stage training incurs a larger drop ($-6.8$ pp on WebArena), indicating that jointly optimizing the imitation reward $R_{\text{sim}}$ and $R_{\text{task}}$ cannot match Mem-π's staged optimization. We attribute this to a mismatch between the two rewards: $R_{\text{sim}}$ encourages imitation of reference memories, whereas $R_{\text{task}}$ rewards memories that improve task success. Since useful memories for new tasks may differ from the references, optimizing both rewards in a single stage can produce conflicting gradients.

表2 的结果表明，两个训练阶段都不可或缺，且统一训练导致的性能下降最大。 移除阶段 1 初始化后，WebArena 下降 5.2 个百分点，说明缺乏初始化良好的记忆分布时，在线 RL 难以收敛。 统一的单阶段训练导致更大的下降（WebArena 上下降 6.8 个百分点），表明联合优化模仿奖励 $R_{\text{sim}}$ 与 $R_{\text{task}}$ 无法达到 Mem-π 分阶段优化的效果。 我们将其归因于两种奖励之间的不匹配：$R_{\text{sim}}$ 鼓励模仿参考记忆，而 $R_{\text{task}}$ 奖励能够提升任务成功率的记忆。 由于对新任务有用的记忆可能不同于参考记忆，在同一阶段优化两种奖励可能产生相互冲突的梯度。

RQ2: Does Stage-2 decision–content policy optimization help? We design three variants targeting its individual components. (i) w/o structured rollout reverts to vanilla GRPO without paired abstain–generate branches; (ii) w/o $\mathrm{\Delta }$-gating replaces gated fusion in Eq. 6 with a naive sum of decision- and content-level advantages; (iii) w/o $R_m$ drops the length-aware memory-quality reward. Variant (i) shows that structured counterfactual sampling is the most critical Stage-2 component. Removing structured rollout costs 4.8 pp on WebArena and 4.5 pp on ALFWorld, the largest drops among RL-objective ablations. Without paired generate-abstain comparisons, most sampling groups lack a routing signal, limiting the policy’s ability to learn when to generate memory. Variants (ii) and (iii) show that $\mathrm{\Delta }$-gating and $R_m$ contribute complementary signals.

RQ2：阶段 2 的决策与内容策略优化有帮助吗？ 我们针对其各个组成部分设计了三种变体。 （i）移除结构化轨迹采样 恢复为普通 GRPO，不再配对弃权与生成分支；（ii）移除 $\mathrm{\Delta }$ 门控 用决策级与内容级优势的简单相加，替换 式（6） 中的门控融合；（iii）移除 $R_m$ 去掉长度感知的记忆质量奖励。 变体 （i） 表明，结构化反事实采样是阶段 2 最关键的组成部分。 移除结构化轨迹采样会使 WebArena 下降 4.8 个百分点、ALFWorld 下降 4.5 个百分点，是 RL 目标消融中最大的降幅。 缺少成对的生成—弃权比较时，多数采样组没有路由信号，限制了策略学习 何时 生成记忆的能力。 变体 （ii） 与 （iii） 表明，$\mathrm{\Delta }$ 门控与 $R_m$ 提供互补的信号。

表2：WebArena 和 ALFWorld 上的消融结果（SR，%）。下标表示相对于完整模型的降幅。

Variant	WA	ALF
Mem-π (Full)	43.1	91.6
Training Stages
w/o Stage 1 init	37.9-5.2	86.9-4.7
Unified single-stage	36.3-6.8	85.7-5.9
RL Objective
w/o structured rollout	38.3-4.8	87.1-4.5
w/o Δ-gating	41.3-1.8	89.6-2.0
w/o Rm	41.9-1.2	90.5-1.1

图3：在 WebArena 上，Mem-π 使用与不使用视觉观测时的性能比较。

Removing $\mathrm{\Delta }$-gating causes consistent drops on both WebArena ($-1.8$ pp) and ALFWorld ($-2.0$ pp), suggesting that naive fusion dilutes the optimization signal by allowing content-level updates even when abstention is preferable. Removing $R_m$ leads to mild degradation, potentially indicating that the length regularizer encourages concise memory and helps reduce noise in generated guidance.

移除 $\mathrm{\Delta }$ 门控后，WebArena（下降 1.8 个百分点）和 ALFWorld（下降 2.0 个百分点）均出现稳定的性能下降，说明简单融合允许在弃权更优时仍执行内容级更新，从而稀释优化信号。 移除 $R_m$ 会导致轻微的性能下降，这可能说明长度正则项鼓励简洁的记忆，并有助于减少生成指导中的噪声。

RQ3: Do visual observations improve memory generation? In addition to textual states, the web nativation tasks provide multimodal observations such as screenshots. We therefore build a vision-language variant of Mem-π using Qwen2.5-VL-7B-Instruct as the memory-policy backbone, and compare it with a text-only variant, i.e., Qwen2.5-7B-Instruct, on WebArena sub-domains.

RQ3：视觉观测是否改善记忆生成？ 除了文本状态，网页导航任务还提供截图等多模态观测。 因此，我们使用 Qwen2.5-VL-7B-Instruct 作为记忆策略骨干，构建 Mem-π 的视觉语言变体，并在 WebArena 的各子领域中，将其与纯文本变体 Qwen2.5-7B-Instruct 比较。

Our experiments show that visual observations provide consistent gains, with larger benefits in visually grounded domains. Figure 3 shows that the multimodal variant outperforms its text-only counterpart across all WebArena sub-domains, improving overall SR by 2.7 pp. The gain is largest in CMS ($+3.8$ pp) and Shopping ($+3.3$ pp), where page layout and product images provide grounding signals difficult to capture from text alone. In contrast, GitLab shows the smallest gain ($+0.9$ pp), consistent with its code-centric content where visual rendering provides limited additional signal.

实验表明，视觉观测带来稳定收益，且在依赖视觉信息的领域中收益更大。 图3 显示，多模态变体在 WebArena 的全部子领域上均优于纯文本变体，整体成功率提升了 2.7 个百分点。 CMS（+3.8 个百分点）和 Shopping（+3.3 个百分点）的收益最大，因为页面布局和商品图像提供了难以仅从文本中捕获的定位依据。 相比之下，GitLab 的收益最小（+0.9 个百分点），这与其以代码为核心、视觉渲染带来的额外信息有限相一致。

3.3. In-Depth Analysis

RQ4: How does adaptive abstention improve performance? We further analyze how adaptive abstention contributes to performance gains by examining the relationship between the model's abstention behavior and task difficulty on the WebArena benchmark. Tasks are grouped into five equal-width bins according to the base agent's success rate, where a lower success rate indicates higher task difficulty. For each bin, we report Mem-π's average abstention rate and the corresponding success-rate improvement over the base agent.

RQ4：自适应弃权如何改善性能？ 我们在 WebArena 基准上考察模型弃权行为与任务难度的关系，进一步分析自适应弃权如何带来性能收益。 根据基础智能体的成功率，将任务划入五个等宽区间；成功率越低，表示任务难度越高。 对于每个区间，我们报告 Mem-π 的平均弃权率，以及相对基础智能体的成功率提升。

图4：在 WebArena 的不同任务难度区间中，弃权率（折线，左轴）以及相对基础智能体的成功率提升（柱形，右轴）。

表3：跨智能体迁移结果（SR，%）。下标表示相对于基础智能体的提升。

Method	Qwen2.5-7B		GPT-5.4-mini
	WA	ALF	WA	ALF
Base Agent	27.9	72.9	27.1	85.3
+ RAG	32.1+4.2	75.0+2.1	31.4+4.3	87.1+1.8
+ Mem-π	46.1+18.2	84.7+11.8	43.1+16.0	91.6+6.3

图5：WebArena 上不同方法的性能与记忆 token 用量之间的关系。

Mem-π abstains on easy tasks, generates on hard tasks, and improves performance most where memory is needed. As shown in Figure 4, abstention is strongly correlated with task difficulty. On the easiest tasks (base-agent SR 80–100%), Mem-π abstains in approximately 71% of cases, while on the hardest tasks, the abstention rate drops to around 13%.

Mem-π 在简单任务上弃权，在困难任务上生成，并在最需要记忆的地方带来最大提升。 如 图4 所示，弃权与任务难度高度相关。 在最简单的任务上（基础智能体成功率为 80%–100%），Mem-π 在约 71% 的情况下弃权；而在最困难的任务上，弃权率降至约 13%。

The SR-improvement bars show the complementary trend: improvement peaks at $+9.7$ pp on the hardest bin and drops to $+1.3$ pp on the easiest bin. This suggests calibrated rather than conservative abstention: Mem-π avoids unnecessary generation when the base agent is already likely to succeed, while generating memory when it provides meaningful benefit.

成功率提升的柱形图呈现出互补趋势：最难区间的提升达到峰值 9.7 个百分点，最易区间则降至 1.3 个百分点。 这说明弃权行为经过了校准，而非一味保守：当基础智能体本来就很可能成功时，Mem-π 避免不必要的生成；当记忆能够带来实质收益时，则生成记忆。

RQ5: Does adaptive memory generalize across agents? We compare RAG and Mem-π using the training-time agent (Qwen2.5-7B-Instruct) and an unseen frontier agent (GPT-5.4-mini), without retraining the memory model.

RQ5：自适应记忆能否跨智能体泛化？ 我们不重新训练记忆模型，分别使用训练时的智能体（Qwen2.5-7B-Instruct）和未见过的前沿智能体（GPT-5.4-mini），比较 RAG 与 Mem-π。

Mem-π generalizes to the unseen GPT-5.4-mini, though gains shrink as the base agent becomes stronger. As shown in Table 3, Mem-π achieves the strongest transfer on the training-time agent (Qwen2.5-7B-Instruct on WebArena: $+18.2$ vs. $+4.2$ pp over RAG), while retaining a clear advantage with GPT-5.4-mini near the capability ceiling (ALFWorld: $+6.3$ vs. $+1.8$ pp). Overall, Mem-π yields $3$–$5\times$ larger gains than RAG. This suggests that memory learned from a weaker agent can encode task guidance at a sufficiently explicit level to remain useful for stronger unseen agents. Training with even weaker agents may further improve interpretability, but could also make Stage 2 RL rewards sparser, introducing a trade-off for future exploration.

Mem-π 能泛化到未见过的 GPT-5.4-mini，但随着基础智能体变强，收益会缩小。 如 表3 所示，Mem-π 在训练时的智能体上实现最强的迁移效果（Qwen2.5-7B-Instruct 在 WebArena 上提升 18.2 个百分点，而 RAG 提升 4.2 个百分点）；在接近能力上限的 GPT-5.4-mini 上仍保有明显优势（ALFWorld 上分别提升 6.3 和 1.8 个百分点）。 总体而言，Mem-π 带来的提升为 RAG 的 3–5 倍。 这表明，从较弱智能体中学得的记忆，能够以足够明确的形式编码任务指导，从而对更强的未见智能体仍然有用。 使用更弱的智能体进行训练或许能进一步提升可解释性，但也可能使阶段 2 的 RL 奖励更加稀疏，形成一个值得未来探索的权衡。

RQ6: Is adaptive generative memory token-efficient? We compare task success rate against the average number of memory tokens prepended to the agent input. For external-memory baselines, this is the length of the retrieved memory content; for generative-memory methods, it is the length of the generated memory. For Mem-π, abstention contributes zero tokens.

RQ6：自适应生成式记忆是否具有 token 效率？ 我们比较任务成功率与添加到智能体输入前的平均记忆 token 数量。 对于外部记忆基线，这指检索到的记忆内容长度；对于生成式记忆方法，这指生成记忆的长度。 对于 Mem-π，弃权不增加任何 token。

Mem-π achieves the best performance-efficiency tradeoff. Figure 5 shows that Mem-π uses 138 memory tokens per task on average, 31% fewer than Stage 1 (200 tokens) and 38% fewer than Memory-R1 (225 tokens), while also attaining the highest success rate, improving Stage 1 from 35.0% to 43.1%. This shows that always generating memory is not only inefficient but can be counterproductive: unnecessary memory adds noise to already solvable tasks. By learning when to abstain, Mem-π reduces memory-token overhead and improves task success, yielding a better performance-efficiency tradeoff.

Mem-π 实现了最佳的性能与效率权衡。 图5 显示，Mem-π 平均每个任务使用 138 个记忆 token，比阶段 1（200 个 token）减少 31%，比 Memory-R1（225 个 token）减少 38%；与此同时，它还取得最高成功率，将阶段 1 的 35.0% 提升至 43.1%。 这表明，始终生成记忆不仅效率低下，还可能适得其反：不必要的记忆会为本已能够解决的任务引入噪声。 通过学习何时弃权，Mem-π 减少了记忆 token 开销并提高了任务成功率，实现了更好的性能与效率权衡。

RQ7: Why does adaptive memory outperform retrieval-based memory? We conduct case analysis of the base agent, RAG, and Mem-π on the WebArena. Figure 6 presents a success-set Venn diagram over these methods and highlights two patterns where Mem-π succeeds and RAG fails.

RQ7：为什么自适应记忆优于基于检索的记忆？ 我们在 WebArena 上对基础智能体、RAG 与 Mem-π 进行案例分析。 图6 展示了这些方法成功任务集合的维恩图，并重点说明 Mem-π 成功而 RAG 失败的两种模式。

Generation adapts retrieved guidance to the current query. When a query specifies a count, identifier, or format that differs from the most similar memory bank entry, retrieval copies the source verbatim, while generation rewrites it from parametric knowledge to match the current input. Case 1 in Figure 6 illustrates this: the retrieved source is a top-$2$ task while the query is top-$3$, so RAG's hint reads “read the first two rows”, whereas Mem-π conditions on the “$3$” and produces “take the first three rows”. Generation resolves the mismatch by rewriting numbers, keys, and formats to fit the query rather than copying from the stored example.

生成能够使检索指导适应当前查询。 当查询指定的数量、标识符或格式不同于记忆库中最相似的条目时，检索会逐字复制源内容，而生成则依据参数化知识改写内容，使其匹配当前输入。 图6 中的案例 1 展示了这一点：检索来源是 top-2 任务，而查询要求 top-3，因此 RAG 的提示写着“读取前两行”，Mem-π 则以查询中的“3”为条件，生成“取前三行”。 生成通过改写数字、键和格式来适配查询，而非复制存储的示例，从而解决这种不匹配。

Abstention discards guidance that no longer applies. When a retrieved hint encodes assumptions, such as a specific product family or identifier, that do not transfer to the current query, any non-empty hint inherits this bias and misleads the agent. Case 2 in Figure 6 illustrates this: the retrieved source narrows the search to “game card case”, but the query asks for the best storage option that fits $40$ cards. Mem-π emits [ABSTAIN], letting the base agent search broadly, while injecting the narrowed hint would have caused failure. Additional cases are provided in Appendix A.4.

弃权会舍弃不再适用的指导。 当检索提示包含无法迁移到当前查询的假设，例如特定商品类别或标识符时，任何非空提示都会继承这种偏向并误导智能体。 图6 中的案例 2 展示了这一点：检索来源将搜索限定为“游戏卡收纳盒”，但查询要求的是能容纳 40 张卡的 最佳 收纳方案。 Mem-π 输出 [ABSTAIN]，让基础智能体进行广泛搜索，而注入这一范围受限的提示则会导致失败。 更多案例见 附录 A.4。

案例 1：热门搜索词（CMS）

任务：“列出我的商店中排名前 3 的搜索词。”

RAG：✗ “……找到 ‘Top Search Terms’ 表……读取前两行，报告每个 ‘Search Term’ 及其 ‘Uses’ 次数。”

Mem-π：✓ “……取 ‘Search Term’ 列的前三行，仅输出这些搜索词名称……”

原因：检索提示来自一个 top-2 源任务，逐字搬用了其中的“2”。生成以查询中明确的“3”为条件，改写了数量。

案例 2：收纳方案搜索（Shopping）

任务：“……找到能容纳全部 40 张［Switch 游戏］卡的最佳收纳方案。”

RAG：✗ “……搜索 ‘game card case’ 或 ‘cartridge holder’，打开标有 40 个以上卡槽的商品……”

Mem-π：✓ [ABSTAIN]

原因：查询要求最佳收纳方案。检索提示将搜索限定在一个狭窄的商品类别，排除了更广泛的收纳产品。弃权让基础智能体恢复开放式搜索。

图6：Mem-π、RAG 与基础智能体的案例分析。左：三种方法成功任务集合的维恩图，彩色数字标出了 Mem-π 相对 RAG 的两种取胜模式：仅 Mem-π 解决的 19 个任务，以及基础智能体与 Mem-π 均解决而 RAG 未解决的 10 个任务。右：每种模式各抽取一个案例，在相同任务查询下比较 Mem-π 与 RAG。

4. Related Work

Learning-based agent memory. Agent memory systems have evolved from static pipelines toward learned memory operations jointly optimized with downstream task outcomes. One line of work distills raw interaction trajectories into structured knowledge including rules, guidelines, or strategies that are retrieved at inference time. For example, AutoGuide compresses offline interaction logs into concise, context-conditional guidelines for web navigation, while ReasoningBank distills generalizable reasoning strategies from both successes and failures, enabling memory-aware test-time scaling. Another line of work optimizes memory operations end-to-end through reinforcement learning, training controllers to manage storage, retrieval, update, and deletion, or co-evolving memory architectures with the agent's policy.

基于学习的智能体记忆。 智能体记忆系统已从静态流程演进为与下游任务结果联合优化的学习式记忆操作。 一类工作将原始交互轨迹蒸馏为规则、指南或策略等结构化知识，并在推理时检索。 例如，AutoGuide 将离线交互日志压缩为简洁、依赖上下文的网页导航指南；ReasoningBank 则从成功与失败案例中蒸馏可泛化的推理策略，实现感知记忆的测试时扩展。 另一类工作通过强化学习端到端地优化记忆操作，训练控制器管理存储、检索、更新与删除，或让记忆架构与智能体策略共同演化。

Memory-R1 and Mem-$\alpha$ equips an LLM with a dedicated manager that learns structured memory operations through outcome-driven RL. More recently, SkillRL builds a hierarchical skill bank that co-evolves through recursive evolution, and MemEvolve goes further by meta-evolving the memory architecture itself across task distributions. Despite these advances, all remain retrieval-centric: they improve when and how to access stored entries, but the memory content itself is fixed at write time. Mem-π departs from this paradigm by modeling memory as a generative policy that dynamically constructs task-adaptive guidance from parametric knowledge.

Memory-R1 和 Mem-α 为 LLM 配备专门的管理器，通过结果驱动的 RL 学习结构化记忆操作。 更近的 SkillRL 构建了通过递归演化共同发展的分层技能库，而 MemEvolve 更进一步，跨任务分布对记忆 架构 本身进行元演化。 尽管取得这些进展，所有方法仍以检索为中心：它们改进了 何时 与 如何 访问已存储条目，但记忆内容本身在写入时就已固定。 Mem-π 跳出这一范式，将记忆建模为生成式策略，根据参数化知识动态构造适应当前任务的指导。

Generative memory. Compared with retrieval-based memory that stores and retrieves external entries, generative memory encodes experience into model parameters and produces useful information on demand. One line of work develops parametric memory modules that internalize retrieval behavior into learnable parameters. Early studies compress long contexts or external knowledge into model parameters, while more recent methods such as MemoryDecoder and MLP Memory train lightweight networks to imitate non-parametric retrievers. CoMEM further extends this direction to multimodal settings, showing that a vision-language model can serve as its own memory encoder.

生成式记忆。 与存储并检索外部条目的检索式记忆相比，生成式记忆将经验编码到模型参数中，并按需生成有用信息。 一类工作构建参数化记忆模块，将检索行为内化为可学习参数。 早期研究将长上下文或外部知识压缩到模型参数中，而 MemoryDecoder、MLP Memory 等较新方法则训练轻量网络来模仿非参数检索器。 CoMEM 进一步将这一方向扩展到多模态设置，表明视觉语言模型可以充当自身的记忆编码器。

R${}^3$Mem bridges memory retention and retrieval through reversible context compression, improving the faithfulness and usability of parametric memory. In agent settings, ParamMem encodes cross-sample reflection patterns into model parameters, enabling diverse and transferable reflection generation. MemGen constructs latent token sequences as machine-native memory through a learned memory trigger and weaver. Most closely related, SEAM trains a experience adapter with GRPO to generate utility-optimized experience entries for a frozen executor. These methods show the promise of parameterized memory generation, but largely treat generation as retrieval imitation or an always-on auxiliary process. Mem-π extends this direction by modeling memory as an adaptive generative policy for multi-step agent interactions, learning both when to generate guidance and what guidance to generate from downstream agent outcomes.

R³Mem 通过可逆上下文压缩连接记忆保留与检索，提高参数化记忆的忠实性与可用性。 在智能体场景中，ParamMem 将跨样本反思模式编码到模型参数中，实现多样化且可迁移的反思生成。 MemGen 通过学习得到的记忆触发器和编织器，构造潜在 token 序列作为机器原生记忆。 与本文最接近的 SEAM 使用 GRPO 训练经验适配器，为冻结的执行器生成经过效用优化的经验条目。 这些方法展示了参数化记忆生成的潜力，但大多将生成视为检索模仿，或始终开启的辅助过程。 Mem-π 进一步将记忆建模为面向多步智能体交互的自适应生成式策略，依据下游智能体结果，同时学习 何时 生成指导以及生成 什么 指导。

5. Conclusion

We presented Mem-π, a framework that formulates agent memory as a generative policy ${\pi }_{\text{mem}}$ rather than retrieval over explicit memory entries. Through experience distillation and adaptation distillation, Mem-π internalizes reusable behavioral knowledge into model parameters and further refines it with downstream task rewards. To jointly optimize when to generate memory and what guidance to generate, we introduced a decision-content decoupled RL objective that separates routing from content optimization through structured counterfactual advantages and per-token credit assignment. Experiments across web navigation, terminal tool use, and embodied environments show that adaptive memory generation consistently improves over retrieval-based and prior RL-optimized memory baselines, establishing generative memory as an effective alternative to retrieval-centric memory systems for LLM agents. Future work can extend Mem-π toward closed-loop memory learning, where agents continuously collect new experiences, update their parametric memory. Another promising direction is grounded and attributable parametric memory, enabling generated guidance to be traced back to supporting experiences while preserving the flexibility of generative memory.

我们提出 Mem-π，将智能体记忆表述为生成式策略 ${\pi }_{\text{mem}}$，而非对显式记忆条目进行检索。 通过经验蒸馏与适应蒸馏，Mem-π 将可复用的行为知识内化到模型参数中，并利用下游任务奖励进一步优化这些知识。 为联合优化 何时 生成记忆与生成 什么 指导，我们提出决策与内容解耦的 RL 目标，通过结构化反事实优势和逐 token 信用分配，将路由与内容优化分离。 网页导航、终端工具使用和具身环境中的实验表明，自适应记忆生成持续优于基于检索和既有的经 RL 优化的记忆基线，确立了生成式记忆作为 LLM 智能体中检索中心式记忆系统的有效替代方案。 未来工作可以将 Mem-π 扩展为闭环记忆学习，使智能体持续收集新经验并更新其参数化记忆。 另一个有前景的方向是有依据且可归因的参数化记忆，使生成的指导能够追溯到支撑它的经验，同时保留生成式记忆的灵活性。