---

LatentMem: Customizing Latent Memory for Multi-Agent Systems

MemoryAgentMulti-Agent40+同济大学上海人工智能实验室香港中文大学南京大学上海交通大学

Fu M, Zhang G, Xue X, et al. LatentMem: Customizing Latent Memory for Multi-Agent Systems[J]. arXiv preprint arXiv:2602.03036v2, 2026.

https://github.com/KANABOON1/LatentMem

https://huggingface.co/Kana-s/LatentMem-Qwen3-4B

---

LatentMem：面向多智能体系统的定制化潜在记忆

Abstract

Large language model (LLM)-powered multi-agent systems (MAS) demonstrate remarkable collective intelligence, wherein multi-agent memory serves as a pivotal mechanism for continual adaptation. However, existing multi-agent memory designs remain constrained by two fundamental bottlenecks: (i) memory homogenization arising from the absence of role-aware customization, and (ii) information overload induced by excessively fine-grained memory entries. To address these limitations, we propose LatentMem, a learnable multi-agent memory framework designed to customize agent-specific memories in a token-efficient manner. Specifically, LatentMem comprises an experience bank that stores raw interaction trajectories in a lightweight form, and a memory composer that synthesizes compact latent memories conditioned on retrieved experience and agent-specific contexts. Further, we introduce Latent Memory Policy Optimization (LMPO), which propagates task-level optimization signals through latent memories to the composer, encouraging it to produce compact and high-utility representations. Extensive experiments across diverse benchmarks and mainstream MAS frameworks show that LatentMem achieves a performance gain of up to $19.36\mathrm{\%}$ over vanilla settings and consistently outperforms existing memory architectures, without requiring any modifications to the underlying frameworks.

由大语言模型（LLM）驱动的多智能体系统（MAS）展现出卓越的集体智能，其中多智能体记忆是实现持续适应的关键机制。 然而，现有多智能体记忆设计仍受两个根本瓶颈限制：(i) 记忆同质化，源于缺少角色感知的定制；以及 (ii) 信息过载，由过于细粒度的记忆条目引起。 为解决这些局限，我们提出 LatentMem，这是一种可学习的多智能体记忆框架，旨在以节省 token 的方式定制智能体专属记忆。 具体而言，LatentMem 包含一个以轻量形式存储原始交互轨迹的经验库，以及一个根据检索到的经验和智能体特定上下文合成紧凑潜在记忆的记忆合成器。 此外，我们提出潜在记忆策略优化（LMPO），它通过潜在记忆把任务级优化信号传播至合成器，促使其生成紧凑且高效用的表示。 在多种基准和主流 MAS 框架上的大量实验表明，LatentMem 相比无记忆设置最高可提升 $19.36\mathrm{\%}$，并且无需修改底层框架即可持续优于现有记忆架构。

1. Introduction

Large Language Model (LLM)-powered multi-agent systems (MAS), have emerged as a powerful framework for solving complex tasks by allowing agents to collaborate or compete beyond the capabilities of individual LLM agents. Pivotal to this success is the concept of multi-agent memory, which enables agents to accumulate, retain, and reuse experiences through interactions with both other agents and the environment, thereby supporting more coherent coordination and continual adaptation.

由大语言模型（LLM）驱动的多智能体系统（MAS）已经成为解决复杂任务的强大框架，它允许智能体通过协作或竞争实现超越单个 LLM 智能体的能力。 这一成功的关键是多智能体记忆概念，它使智能体能够通过与其他智能体及环境交互来积累、保留和复用经验，从而支持更连贯的协调与持续适应。

图1：现有多智能体记忆与 LatentMem 的范式比较。LatentMem 不依赖手工设计的记忆单元，而是结合原始轨迹与智能体画像，从潜在空间提取智能体专属记忆。

Building on this memory foundation, recent studies have increasingly explored multi-granularity memory repositories that capture experiences at different levels of abstraction, including (i) MAS trajectories, (ii) distilled semantic insights, and (iii) orchestrable skill schemas. These designs endow memory systems with the ability to capture diverse memory patterns, such as trajectory summarization and high-level insight extraction, enabling MAS to adaptively integrate past experiences and jointly refine decision-making strategies.

在这一记忆基础上，近期研究越来越多地探索多粒度记忆库，以捕获不同抽象层级的经验，包括 (i) MAS 轨迹、(ii) 提炼后的语义洞见和 (iii) 可编排的技能模式。 这些设计使记忆系统能够捕获轨迹摘要和高层洞见抽取等多种记忆模式，从而使 MAS 能够自适应地整合过去经验，并共同完善决策策略。

However, despite the growing sophistication of existing memory systems, they remain constrained by two key limitations: (i) Memory homogenization: Most methods adopt a one-size-fits-all strategy, ignoring the functional heterogeneity of agents, which undermines role adherence and amplifies correlated errors, weakening system robustness and hindering long-term adaptation. (ii) Information overload: MAS inherently involves long interaction contexts, and multi-granularity memory designs further amplify this burden by introducing large volumes of stored entries, ultimately overwhelming agents and obscuring critical decision signals. Given the aforementioned challenges, a natural question arises: Given long and complex contexts in MAS, can we design a learnable memory that is both role-aware and token-efficient, without extensive manual engineering?

然而，尽管现有记忆系统日益复杂，它们仍受两个关键局限约束： (i) 记忆同质化： 大多数方法采用一刀切策略，忽视智能体之间的功能异质性，这会削弱角色遵循、放大相关错误、降低系统鲁棒性，并阻碍长期适应。 (ii) 信息过载： MAS 本身就涉及较长的交互上下文，而多粒度记忆设计又通过引入大量存储条目进一步加重这一负担，最终使智能体不堪重负，并掩盖关键决策信号。 鉴于上述挑战，一个自然的问题随之产生： 面对 MAS 中漫长而复杂的上下文，我们能否在无需大量人工设计的情况下，构建一种既能感知角色又节省 token 的可学习记忆？

To address these challenges, we propose LatentMem, a latent multi-agent memory framework that materializes agent-aware memory customization via token-efficient latent memory generation. Specifically, LatentMem consists of two components: a lightweight experience bank for storing and retrieving raw MAS trajectories, and a memory composer that leverages agent profiles to distill raw trajectories into role-aware, compact latent memories and integrate them into the agents’ reasoning process. To encourage the memory composer to distill transferable, high-utility latent representations from raw trajectories, we propose Latent Memory Policy Optimization (LMPO), which computes advantages from relative rewards within multi-agent rollouts, optimizes token-level objectives, and exploits latent memory differentiability to enable gradient backpropagation through the memory composer.

为解决这些挑战，我们提出 LatentMem，这是一种潜在多智能体记忆框架，通过节省 token 的潜在记忆生成实现智能体感知的记忆定制。 具体而言，LatentMem 包含两个组件：用于存储和检索原始 MAS 轨迹的轻量级经验库，以及利用智能体画像把原始轨迹提炼为角色感知的紧凑潜在记忆，并将其整合进智能体推理过程的记忆合成器。 为促使记忆合成器从原始轨迹中提炼可迁移、高效用的潜在表示，我们提出潜在记忆策略优化（LMPO）；它根据多智能体 rollout 中的相对奖励计算优势，优化 token 级目标，并利用潜在记忆的可微性，让梯度能够通过记忆合成器反向传播。

As a novel attempt in latent MAS memory, LatentMem offers three principal advantages: (I) It conditions the memory composer on agent role profiles to customize role-aware latent memories, thereby mitigating memory homogenization; (II) It encodes multi-agent memory as fixed-length latent representations rather than unbounded discrete textual traces, thereby mitigating information overload; (III) It exploits LMPO and latent memory differentiability to enable autonomous memory internalization and reconstruction, thereby avoiding language constraints and obviating the need for meticulously engineered memory architectures.

作为潜在 MAS 记忆方向的一次新尝试，LatentMem 具有三项主要优势： (I) 它以智能体角色画像为条件驱动记忆合成器，定制角色感知的潜在记忆，从而缓解记忆同质化； (II) 它把多智能体记忆编码为固定长度的潜在表示，而非无界的离散文本轨迹，从而缓解信息过载； (III) 它利用 LMPO 和潜在记忆的可微性，实现自主的记忆内化与重构，从而摆脱语言约束，也不再需要精心设计的记忆架构。

Extensive experiments across six benchmarks and four mainstream MAS frameworks demonstrate that LatentMem achieves: (I) high performance, improving state-of-the-art MAS by up to $16.20\mathrm{\%}$ and $18.45\mathrm{\%}$ in knowledge QA and code generation tasks, respectively; (II) high efficiency, using $50\mathrm{\%}$ fewer tokens and reducing inference time to $\sim 2/3$ compared to mainstream memory designs; and (III) strong generalization, with out-of-domain datasets such as PDDL showing a $7.10\mathrm{\%}$ improvement, and unseen MAS such as CAMEL exhibiting a $7.90\mathrm{\%}$ gain compared to the vanilla setting. These results establish LatentMem as a novel and effective framework for MAS memory.

在六个基准和四种主流 MAS 框架上的大量实验表明，LatentMem 实现了： (I) 高性能，在知识问答和代码生成任务上分别使最先进 MAS 最高提升 $16.20\mathrm{\%}$ 和 $18.45\mathrm{\%}$； (II) 高效率，与主流记忆设计相比减少 $50\mathrm{\%}$ 的 token 使用，并把推理时间降至约 $2/3$；以及 (III) 强泛化能力，在 PDDL 等域外数据集上提升 $7.10\mathrm{\%}$，并在 CAMEL 等未见 MAS 上相比无记忆设置提升 $7.90\mathrm{\%}$。 这些结果确立了 LatentMem 作为一种新颖且有效的 MAS 记忆框架的地位。

2. Related Works

LLM-Based Multi-Agent Systems. MAS is a framework in which multiple agents collaborate by assuming distinct roles to achieve shared objectives. Our work focuses on leveraging MAS for task-specific problem solving. Early task-solving MAS frameworks typically rely on predefined, static workflows, while more recent studies have enabled MAS to dynamically reconfigure their organizational structures, improving adaptability to diverse and complex tasks while reducing computational costs. However, these methods typically require extensive searches over the design space, resulting in considerable computational and token overhead. Consequently, developing a lightweight mechanism for flexible MAS adaptation remains an open and challenging research problem.

基于 LLM 的多智能体系统。 MAS 是一种让多个智能体承担不同角色、通过协作实现共同目标的框架。 我们的工作聚焦于利用 MAS 解决特定任务。 早期的任务求解 MAS 框架通常依赖预定义的静态工作流，而近期研究则使 MAS 能够动态重构其组织结构，在降低计算成本的同时提高对多样复杂任务的适应性。 然而，这些方法通常需要在设计空间中进行大量搜索，带来可观的计算与 token 开销。 因此，为 MAS 的灵活适应开发一种轻量级机制，仍是一个开放且具有挑战性的研究问题。

Memory in Multi-Agent Systems. Memory enables agents to accumulate experience through interactions, thereby supporting coherent coordination and continual adaptation. It plays a crucial role in task-solving and social simulation; our focus lies primarily on the former. Early memory designs in MAS typically rely on simple, within-trial mechanisms coupled to the system itself, such as a shared pool storing raw trajectories. Modern memories, by contrast, have shifted towards more intricate and flexible structures. Representative examples include OAgents, which employs multi-granularity memory; EvolveR and Agent KB, which compress raw trajectories into high-level semantic units; and MIRIX, which transforms user goals into orchestrable procedural memories. However, these approaches overlook heterogeneous, role-aware memory design. LatentMem addresses this limitation by equipping each agent with a compact, role-aware latent memory, thereby reinforcing role compliance, enhancing coordination, and improving continual adaptation.

多智能体系统中的记忆。 记忆使智能体能够通过交互积累经验，从而支持连贯协调和持续适应。 它在任务求解和社会模拟中发挥关键作用；我们的重点主要是前者。 早期 MAS 记忆设计通常依赖与系统本身耦合的简单单次试验内机制，例如用于存储原始轨迹的共享池。 相比之下，现代记忆已经转向更复杂、更灵活的结构。 代表性示例包括采用多粒度记忆的 OAgents；把原始轨迹压缩为高层语义单元的 EvolveR 和 Agent KB；以及把用户目标转换为可编排程序性记忆的 MIRIX。 然而，这些方法忽视了异质、角色感知的记忆设计。 LatentMem 为每个智能体配备紧凑的角色感知潜在记忆，从而强化角色遵循、增强协调并改善持续适应，解决了这一局限。

图2：LatentMem 概览。框架依次执行：（1）从经验库检索相关轨迹；（2）通过经 LMPO 训练的记忆合成器，把轨迹与智能体角色画像压缩为潜在记忆；（3）在不修改智能体架构的情况下，把这些记忆注入智能体推理过程；（4）存储新轨迹以实现持续改进。

3. Preliminary

Notations. Consider a multi-agent system $\mathcal{X}$ containing $N$ agents $\mathcal{A}=\{a_1,a_2,\dots ,a_N\}$, and equipped with a global memory module $\mathcal{M}$ that stores and retrieves shared information among agents. Formally, the system can be represented as the tuple:

符号。 考虑一个包含 $N$ 个智能体 $\mathcal{A}=\{a_1,a_2,\dots ,a_N\}$ 的多智能体系统 $\mathcal{X}$，它配备全局记忆模块 $\mathcal{M}$，用于存储和检索智能体之间的共享信息。 形式上，该系统可表示为以下元组：

$$\mathcal{X}=(\mathcal{A},\mathcal{G},\mathcal{M}).$$

Each agent $a_k=({\gamma }_k,{\pi }_{{\theta }_k})$ is defined by a role profile ${\gamma }_k$ and a policy ${\pi }_{{\theta }_k}$ parameterized by ${\theta }_k$. During execution, an agent receives an input prompt $p$ and a retrieved memory $m$ from $\mathcal{M}$, and produces a response $o$, denoted as $o=a_k(p,m)$. The execution graph $\mathcal{G}$ governs the topological order in which agents operate. Depending on the system architecture, $\mathcal{G}$ can be instantiated as either a static predefined topology or a centralized dynamic regulation mechanism.

每个智能体 $a_k=({\gamma }_k,{\pi }_{{\theta }_k})$ 由角色画像 ${\gamma }_k$ 和以 ${\theta }_k$ 为参数的策略 ${\pi }_{{\theta }_k}$ 定义。 执行期间，智能体接收输入提示 $p$ 和从 $\mathcal{M}$ 检索出的记忆 $m$，并生成响应 $o$，记作 $o=a_k(p,m)$。 执行图 $\mathcal{G}$ 控制智能体运行的拓扑顺序。 根据系统架构，$\mathcal{G}$ 可以实例化为静态预定义拓扑，也可以实例化为集中式动态调节机制。

Problem Formulation. Our objective is to find the memory module $\mathcal{M}$ that maximizes the expected performance of MAS $\mathcal{X}$, which is formally defined as:

问题形式化。 我们的目标是找到使 MAS $\mathcal{X}$ 的期望性能最大化的记忆模块 $\mathcal{M}$，其形式化定义如下：

$$\underset{\mathcal{M}}{max}{\mathbb{E}}_{q\sim \mathcal{D},\tau \sim \mathcal{X}(q)}[R(\tau )].$$

where $\mathcal{D}$ denotes the dataset and $q$ is a query sampled from it. The system $\mathcal{X}$ processes the query $q$ to produce a reasoning trajectory $\tau$, and the reward function $R$ extracts the final answer from $\tau$ and evaluates its correctness. This formulation is agnostic to specific memory architectures, ranging from hand-crafted symbolic systems to learnable, parameterized counterparts. Conventional memory systems often rely on predefined patterns to accumulate experiences, while our approach adopts a learnable memory module that generates compact, role-aware latent representations for dynamic integration into each agent's reasoning.

其中，$\mathcal{D}$ 表示数据集，$q$ 是从中采样的查询。 系统 $\mathcal{X}$ 处理查询 $q$ 并生成推理轨迹 $\tau$，奖励函数 $R$ 从 $\tau$ 中提取最终答案并评估其正确性。 这一形式化定义不依赖具体记忆架构，既适用于手工构建的符号系统，也适用于可学习的参数化系统。 传统记忆系统通常依赖预定义模式来积累经验，而我们的方法采用可学习记忆模块，生成紧凑、角色感知的潜在表示，并动态整合进每个智能体的推理过程。

4. Methodology

In this section, we first present the overall pipeline of our proposed LatentMem framework (Section 4.1). Then we detail each module within the framework, including the experience bank (Section 4.2) and the memory composer (Section 4.3). Finally, we introduce Latent Memory Policy Optimization (LMPO), which enables end-to-end optimization of the memory composer through task-level feedback and reinforcement learning algorithm (Section 4.4).

在本节中，我们首先介绍所提出 LatentMem 框架的整体流程（第 4.1 节）。 随后，我们详细说明框架中的各个模块，包括经验库（第 4.2 节）和记忆合成器（第 4.3 节）。 最后，我们介绍潜在记忆策略优化（LMPO），它通过任务级反馈和强化学习算法实现记忆合成器的端到端优化（第 4.4 节）。

4.1 Overall Pipeline

The overall pipeline of our proposed LatentMem framework is illustrated in Figure 2. It consists of two core components: a lightweight experience bank $\mathcal{B}$ for storing and retrieving historical trajectories, and a learnable memory composer $\mathcal{C}$ that transforms the retrieved relevant trajectories into compact, role-aware latent memories.

我们提出的 LatentMem 框架的整体流程如图2所示。 它包含两个核心组件：用于存储和检索历史轨迹的轻量级经验库 $\mathcal{B}$，以及把检索到的相关轨迹转换为紧凑、角色感知潜在记忆的可学习记忆合成器 $\mathcal{C}$。

Upon receiving a new query, LatentMem first retrieves a subset of relevant trajectories from the experience bank. These trajectories capture the interactions and behaviors of agents in previous MAS executions, forming a historical context that can guide reasoning. The retrieved trajectories, together with each agent’s role profile, are then processed by the memory composer, which distills them into compact latent memories tailored to the corresponding agents. During each agent’s reasoning process, these latent memories are appended to the token embeddings as additional latent tokens, forming a memory-augmented representation that incentivizes the agent to leverage prior experience and generate improved outputs. After the MAS completes a task, the newly generated trajectory is appended to the experience bank, enabling incremental accumulation of experiences.

收到新查询后，LatentMem 首先从经验库中检索相关轨迹子集。 这些轨迹捕获智能体在先前 MAS 执行中的交互与行为，形成能够指导推理的历史上下文。 随后，记忆合成器共同处理检索到的轨迹和每个智能体的角色画像，把它们提炼为针对相应智能体定制的紧凑潜在记忆。 在每个智能体的推理过程中，这些潜在记忆作为额外潜在 token 附加到 token 嵌入之后，形成记忆增强表示，促使智能体利用先前经验并生成更好的输出。 MAS 完成任务后，新生成的轨迹会追加到经验库中，从而实现经验的增量积累。

This procedure forms a self-improving loop, allowing LatentMem to continuously refine agent reasoning, support long-horizon coordination, and enhance continual adaptation. Moreover, the injection of latent memories maintains end-to-end differentiability of the entire forward process, facilitating efficient RL-based post-training without incurring the heavy computation of retraining foundation models.

这一过程形成了自我改进循环，使 LatentMem 能够持续完善智能体推理、支持长程协调并增强持续适应。 此外，潜在记忆的注入维持了整个前向过程的端到端可微性，使系统无需承担重新训练基础模型的高昂计算成本，即可进行高效的基于 RL 的后训练。

4.2 Experience Bank

To accurately record historical MAS trajectories for future reuse, we construct an extremely lightweight experience bank $\mathcal{B}$. In line with the principle that scalable systems should rely on general learning mechanisms rather than hand-crafted knowledge, this bank stores and retrieves only raw trajectories, without introducing any human priors such as trajectory condensation or insight extraction,

为准确记录历史 MAS 轨迹以供未来复用，我们构建了一个极其轻量的经验库 $\mathcal{B}$。 遵循可扩展系统应依赖通用学习机制而非手工知识的原则，该经验库只存储和检索原始轨迹，不引入轨迹压缩或洞见抽取等任何人工先验，

Initialization. We populate the experience bank $\mathcal{B}$ with a wide-ranging collection of trajectories covering multiple domains and MAS frameworks to enable the memory composer $\mathcal{C}$ to learn generalizable memory patterns across diverse domains and agent coordination patterns. The resulting initialized bank is denoted as $\{{\tau }_i{\}}_{i=1}^C$, where $C$ specifies its initial capacity. Each trajectory $\tau =\{({\alpha }_j,p_j,o_j){\}}_{j=1}^H$ records, at each step, the index of the active agent ${\alpha }_j$ along with its input prompt $p_j$ and corresponding output $o_j$, where $H$ denotes the trajectory horizon.

初始化。 我们使用覆盖多个领域和 MAS 框架的广泛轨迹集合填充经验库 $\mathcal{B}$，使记忆合成器 $\mathcal{C}$ 能够跨不同领域和智能体协调模式学习可泛化的记忆模式。 所得初始化经验库记作 $\{{\tau }_i{\}}_{i=1}^C$，其中 $C$ 表示其初始容量。 每条轨迹 $\tau =\{({\alpha }_j,p_j,o_j){\}}_{j=1}^H$ 在每一步记录活跃智能体的索引 ${\alpha }_j$、其输入提示 $p_j$ 和对应输出 $o_j$，其中 $H$ 表示轨迹长度。

Retrieval. Upon receiving a new user query $q$, LatentMem performs similarity-based retrieval over $\mathcal{B}$ to obtain a subset of $K$ relevant trajectories ${\mathcal{T}}_q$:

检索。 收到新的用户查询 $q$ 后，LatentMem 在 $\mathcal{B}$ 上执行基于相似度的检索，以获得包含 $K$ 条相关轨迹的子集 ${\mathcal{T}}_q$：

$$\begin{array}{rl}{\mathcal{T}}_q& =\mathrm{top}\text{-}{K}_{{\tau }_i\in \mathcal{B}}(\mathrm{sim}(\mathbf{v}(q),\mathbf{v}({\tau }_i)))\\ & =\{{\tau }_i{\}}_{i=1}^K.\end{array}$$

where $\mathbf{v}(\cdot )$ maps queries or trajectories into a latent embedding space, e.g., using MiniLM, and $\text{sim}(\cdot ,\cdot )$ denotes the cosine similarity. The retrieved trajectories will be subsequently processed by the memory composer, which distills them into latent memories to guide subsequent MAS reasoning tasks.

其中，$\mathbf{v}(\cdot )$ 把查询或轨迹映射到潜在嵌入空间，例如使用 MiniLM；$\text{sim}(\cdot ,\cdot )$ 表示余弦相似度。 随后，记忆合成器会处理检索到的轨迹，把它们提炼为潜在记忆，以指导后续 MAS 推理任务。

Update. Once a task is completed, the new trajectory ${\tau }_{\text{new}}$ is appended to the experience bank $\mathcal{B}$ for future reuse:

更新。 任务完成后，新轨迹 ${\tau }_{\text{new}}$ 会被追加到经验库 $\mathcal{B}$ 中，以供未来复用：

$$\mathcal{B}\leftarrow \mathcal{B}\cup \{{\tau }_{\text{new}}\}.$$

This streamlined update mechanism allows LatentMem to incrementally accumulate experiences online during inference, facilitating continual adaptation and cross-task coordination without the need for retraining. However, directly feeding the retrieved raw trajectories to agents is suboptimal, as it can overwhelm LLMs with excessive context and fails to capture role-specific representations in heterogeneous MAS. To address these limitations, we introduce the memory composer $\mathcal{C}$, which effectively transforms low-level raw trajectories into compact, high-level, role-aware latent memories.

这种精简的更新机制使 LatentMem 能够在推理期间在线增量积累经验，无需重新训练即可促进持续适应和跨任务协调。 然而，直接把检索到的原始轨迹输入智能体并非最优，因为过量上下文可能使 LLM 不堪重负，而且无法捕获异质 MAS 中的角色特定表示。 为解决这些局限，我们引入记忆合成器 $\mathcal{C}$，把低层原始轨迹有效转换为紧凑、高层且角色感知的潜在记忆。

4.3 Memory Composer

After identifying the relevant raw trajectories ${\mathcal{T}}_q$, we introduce the memory composer $\mathcal{C}$, which provides each agent with generalizable memories. Formally, $\mathcal{C}$ is instantiated as a deep neural network ${\sigma }_{\varphi }$ parameterized by $\varphi$. At each reasoning step $j$, ${\sigma }_{\varphi }$ takes as input the retrieved trajectories ${\mathcal{T}}_q$ and the role profile ${\gamma }_{{\alpha }_j}$ of the active agent $a_{{\alpha }_j}$, producing a fixed-length, agent-aware latent memory matrix:

确定相关原始轨迹 ${\mathcal{T}}_q$ 后，我们引入记忆合成器 $\mathcal{C}$，为每个智能体提供可泛化记忆。 形式上，$\mathcal{C}$ 被实例化为以 $\varphi$ 为参数的深度神经网络 ${\sigma }_{\varphi }$。 在每个推理步骤 $j$，${\sigma }_{\varphi }$ 接收检索到的轨迹 ${\mathcal{T}}_q$ 和活跃智能体 $a_{{\alpha }_j}$ 的角色画像 ${\gamma }_{{\alpha }_j}$，生成固定长度、智能体感知的潜在记忆矩阵：

$$m_j={\sigma }_{\varphi }({\gamma }_{{\alpha }_j},{\mathcal{T}}_q)\in {\mathbb{R}}^{L^{\prime }\times D}.$$

where $L^{\prime }$ is a fixed length of the latent memory and $D$ denotes the hidden dimension of the foundation model. To conduct reasoning, the active agent $a_{{\alpha }_j}$ first encodes its input prompt $p_j$ into a sequence of hidden state vectors $h_j=(h_j^{(1)},\dots ,h_j^{(L)})\in {\mathbb{R}}^{L\times D}$. The agent’s latent memory $m_j$ is then concatenated to $h_j$ to form an extended input shaped ${\mathbb{R}}^{(L+L^{\prime })\times D}$, resulting in a memory-augmented policy:

其中，$L^{\prime }$ 是潜在记忆的固定长度，$D$ 表示基础模型的隐藏维度。 为执行推理，活跃智能体 $a_{{\alpha }_j}$ 首先把输入提示 $p_j$ 编码为隐藏状态向量序列 $h_j=(h_j^{(1)},\dots ,h_j^{(L)})\in {\mathbb{R}}^{L\times D}$。 随后，智能体的潜在记忆 $m_j$ 与 $h_j$ 拼接，形成形状为 ${\mathbb{R}}^{(L+L^{\prime })\times D}$ 的扩展输入，由此得到记忆增强策略：

$${\tilde{\pi }}_{{\theta }_{{\alpha }_j}}(p_j,m_j)={\pi }_{{\theta }_{{\alpha }_j}}(\mathrm{concat}(h_j,m_j)).$$

where ${\tilde{\pi }}_{{\theta }_{{\alpha }_j}}$ is a wrapped version of ${\pi }_{{\theta }_{{\alpha }_j}}$ that seamlessly incorporates memory injection at the model level, remaining transparent to the agent layer and requiring no modifications to the system architecture.

其中，${\tilde{\pi }}_{{\theta }_{{\alpha }_j}}$ 是 ${\pi }_{{\theta }_{{\alpha }_j}}$ 的封装版本，它在模型层无缝整合记忆注入，对智能体层保持透明，且无需修改系统架构。

4.4 Latent Memory Policy Optimization (LMPO)

To enable end-to-end optimization of LatentMem while preserving strong generalization across diverse domains and MAS frameworks, we propose Latent Memory Policy Optimization (LMPO), a variant of GRPO, which encourages the memory composer to generate transferable, high-utility latent representations.

为实现 LatentMem 的端到端优化，同时保持其跨不同领域和 MAS 框架的强泛化能力，我们提出潜在记忆策略优化（LMPO）；它是 GRPO 的一种变体，能够促使记忆合成器生成可迁移、高效用的潜在表示。

Parametric Dependency. We first describe the gradient flow during LMPO, in which the learning signal propagates through the latent memories to optimize the memory composer $\mathcal{C}$, while keeping the agent backbones $\{{\theta }_k{\}}_{k=1}^N$ frozen. Formally, given a query $q$ and the retrieved trajectories ${\mathcal{T}}_q$ from the experience bank $\mathcal{B}$, the generation of a new trajectory $\tau =\{({\alpha }_j,p_j,o_j){\}}_{j=1}^H$ is factorized sequentially as:

参数依赖。 我们首先描述 LMPO 中的梯度流：学习信号通过潜在记忆传播，以优化记忆合成器 $\mathcal{C}$，同时冻结智能体骨干 $\{{\theta }_k{\}}_{k=1}^N$。 形式上，给定查询 $q$ 和从经验库 $\mathcal{B}$ 检索出的轨迹 ${\mathcal{T}}_q$，新轨迹 $\tau =\{({\alpha }_j,p_j,o_j){\}}_{j=1}^H$ 的生成按顺序分解为：

$$\begin{array}{rl}& \mathbb{P}(\tau \mid q,{\mathcal{T}}_q;\varphi ,\{{\theta }_k{\}}_{k=1}^N)\\ & =\prod _{j=1}^H\mathbb{P}(o_j\mid p_j,m_j;{\theta }_{{\alpha }_j}).\end{array}$$

Crucially, the latent memory $m_j={\sigma }_{\varphi }({\mathcal{T}}_q,{\gamma }_{{\alpha }_j})$, as defined above, serves as a differentiable interface through which $\varphi$ influences the autoregressively generated output $o_j$ of the active agent $a_{{\alpha }_j}$ at reasoning step $j$:

关键在于，如上所定义的潜在记忆 $m_j={\sigma }_{\varphi }({\mathcal{T}}_q,{\gamma }_{{\alpha }_j})$ 是一个可微接口，$\varphi$ 通过它影响活跃智能体 $a_{{\alpha }_j}$ 在推理步骤 $j$ 自回归生成的输出 $o_j$：

$$\begin{array}{rl}& \mathbb{P}(o_j\mid p_j,m_j;{\theta }_{{\alpha }_j})\\ & =\prod _{t=1}^T{\tilde{\pi }}_{{\theta }_{{\alpha }_j}}(o_j^{(t)}\mid p_j,o_j^{(<t)},m_j).\end{array}$$

Since the composite policy ${\tilde{\pi }}_{{\theta }_{{\alpha }_j}}$ is conditioned on $m_j$, the gradient of any task-level objective can be backpropagated through the agent’s forward pass to refine $\varphi$. This dependency ensures that the memory composer can be optimized end-to-end to produce high-quality latent memories, forming the basis of our policy optimization strategy.

由于复合策略 ${\tilde{\pi }}_{{\theta }_{{\alpha }_j}}$ 以 $m_j$ 为条件，任意任务级目标的梯度都可以通过智能体的前向传播反向传播，以更新 $\varphi$。 这一依赖关系确保记忆合成器可以进行端到端优化，生成高质量潜在记忆，并构成我们策略优化方法的基础。

Policy Optimization. Building on the differentiable path above, LMPO leverages task-level feedback through latent memories as a bridge to directly optimize memory composer attached to the MAS, encouraging it to distill high-utility, agent-specific memories from retrieved raw trajectories and thereby enhance reasoning quality and overall performance. Formally, given a query $q$ and its retrieved relevant trajectories ${\mathcal{T}}_q$, we sample a group of $G$ trajectories:

策略优化。 基于上述可微路径，LMPO 以潜在记忆为桥梁利用任务级反馈，直接优化附加到 MAS 的记忆合成器，促使其从检索到的原始轨迹中提炼高效用、智能体专属的记忆，从而提高推理质量和整体性能。 形式上，给定查询 $q$ 及其检索到的相关轨迹 ${\mathcal{T}}_q$，我们采样一组 $G$ 条轨迹：

$$\{{\hat{\tau }}_i{\}}_{i=1}^G\sim \mathbb{P}(\cdot \mid q,{\mathcal{T}}_q;\varphi ,\{{\theta }_k{\}}_{k=1}^N).$$

Each trajectory is evaluated using reward $R({\hat{\tau }}_i)$, and its relative quality is captured by the group-based advantage:

每条轨迹都使用奖励 $R({\hat{\tau }}_i)$ 进行评估，其相对质量由基于组的优势捕获：

$${\hat{A}}_i=\frac{R({\hat{\tau }}_i)-\mathrm{mean}(\{R({\hat{\tau }}_i){\}}_{i=1}^G)}{\mathrm{std}(\{R({\hat{\tau }}_i){\}}_{i=1}^G)+ϵ}.$$

While standard reinforcement learning often employs trajectory-level objectives, such approaches treat all sequences equally, causing tokens in longer MAS interactions to contribute disproportionately less to the gradient. This makes it difficult for the memory composer to capture critical coordination patterns within long-horizon tasks. Instead, we adopt a token-level surrogate objective:

虽然标准强化学习通常采用轨迹级目标，但这类方法会平等对待所有序列，导致较长 MAS 交互中的 token 对梯度的贡献不成比例地减小。 这使记忆合成器难以捕获长程任务中的关键协调模式。 因此，我们采用 token 级替代目标：

$$\begin{array}{rl}{\mathcal{J}}_{\mathrm{LMPO}}(\varphi )& ={\mathbb{E}}_{q\sim \mathcal{D},\{{\hat{\tau }}_i{\}}_{i=1}^G\sim \mathbb{P}(\cdot \mid q,{\mathcal{T}}_q)}\\ & [\frac{1}{|\{{\hat{\tau }}_i{\}}_{i=1}^G|}\sum _{i,j,t}{\mathcal{L}}_{i,j,t}(\varphi )].\end{array}$$

where $|\{{\hat{\tau }}_i{\}}_{i=1}^G|$ is the total number of generated tokens within the trajectory group and ${\mathcal{L}}_{i,j,t}(\varphi )$ is defined as:

其中，$|\{{\hat{\tau }}_i{\}}_{i=1}^G|$ 是轨迹组内生成 token 的总数，${\mathcal{L}}_{i,j,t}(\varphi )$ 定义为：

$$\begin{array}{rl}{\mathcal{L}}_{i,j,t}(\varphi )=min(& r_{i,j,t}(\varphi ){\hat{A}}_i,\\ & \mathrm{clip}(r_{i,j,t}(\varphi ),1-\epsilon ,1+\epsilon ){\hat{A}}_i).\end{array}$$

and the token-level importance sampling ratio

而 token 级重要性采样比率为

$$\begin{array}{r}{r}_{i,j,t}(\varphi )=\frac{{\tilde{\pi }}_{\theta }(o_{i,j}^{(t)}\mid p_{i,j},o_{i,j}^{(<t)},{\sigma }_{\varphi }({\gamma }_{{\alpha }_{i,j}},{\mathcal{T}}_q))}{{\tilde{\pi }}_{\theta }(o_{i,j}^{(t)}\mid p_{i,j},o_{i,j}^{(<t)},{\sigma }_{{\varphi }_{\mathrm{old}}}({\gamma }_{{\alpha }_{i,j}},{\mathcal{T}}_q))}.\end{array}$$

measures how the policy of agent $a_{{\alpha }_{i,j}}$ at reasoning step $j$ and token $t$ is modulated by the updated memory.

它衡量更新后的记忆如何调节智能体 $a_{{\alpha }_{i,j}}$ 在推理步骤 $j$ 和 token $t$ 处的策略。

表1：在六个基准上比较不同记忆框架的性能，使用 Qwen3-4B-Instruct-2507 作为骨干模型。粗体表示最佳结果，下划线表示次优结果。TriviaQA、KodCode、StrategyQA 和 PopQA 是域内基准，BigCodeBench 和 PDDL 是域外基准；AutoGen 和 MacNet 是训练分布内 MAS 框架，CAMEL 和 DyLAN 是具有新智能体角色与协作模式的未见框架。

Method	TriviaQA	KodCode	StrategyQA	PopQA	BigCodeBench	PDDL	Average
Held-in · AutoGen
No-memory	60.31	68.40	58.25	38.78	79.53	16.39	53.61
ChatDev	57.34 ↓2.97	68.55 ↑0.15	59.18 ↑0.93	33.24 ↓5.54	79.15 ↓0.38	15.22 ↓1.17	52.11 ↓1.50
MetaGPT	60.35 ↑0.04	70.05 ↑1.65	60.92 ↑2.67	33.80 ↓4.98	80.40 ↑0.87	11.95 ↓4.44	52.91 ↓0.70
Generative	59.65 ↓0.66	70.90 ↑2.50	62.66 ↑4.41	40.37 ↑1.59	81.14 ↑1.61	13.94 ↓2.45	54.78 ↑1.17
Voyager	57.50 ↓2.81	68.95 ↑0.55	60.48 ↑2.23	33.56 ↓5.22	79.40 ↓0.13	13.62 ↓2.77	52.25 ↓1.36
G-Memory	60.56 ↑0.25	71.40 ↑3.00	63.89 ↑5.64	42.67 ↑3.89	80.96 ↑1.43	17.06 ↑0.67	56.09 ↑2.48
JoyAgent	59.44 ↓0.87	70.90 ↑2.50	62.33 ↑4.08	41.89 ↑3.11	80.52 ↑0.99	14.26 ↓2.13	54.89 ↑1.28
OAgents	59.85 ↓0.46	70.80 ↑2.40	62.45 ↑4.20	40.70 ↑1.92	81.32 ↑1.79	16.70 ↑0.31	55.30 ↑1.69
LatentMem	76.51 ↑16.20	76.80 ↑8.40	65.48 ↑7.23	52.70 ↑13.92	81.49 ↑1.96	23.49 ↑7.10	62.75 ↑9.14
Held-in · MacNet
No-memory	53.77	70.40	56.44	24.89	78.15	20.73	50.73
ChatDev	57.29 ↑3.52	70.50 ↑0.10	60.22 ↑3.78	35.33 ↑10.44	79.17 ↑1.02	15.85 ↓4.88	53.06 ↑2.33
MetaGPT	61.16 ↑7.39	71.50 ↑1.10	61.14 ↑4.70	33.98 ↑9.09	78.33 ↑0.18	17.81 ↓2.92	53.99 ↑3.26
Generative	59.89 ↑6.12	71.15 ↑0.75	61.79 ↑5.35	43.39 ↑18.50	79.47 ↑1.32	16.81 ↓3.92	55.42 ↑4.69
Voyager	58.19 ↑4.42	69.80 ↓0.60	61.35 ↑4.91	35.38 ↑10.49	80.53 ↑2.38	14.75 ↓5.98	53.33 ↑2.60
G-Memory	62.43 ↑8.66	72.50 ↑2.10	62.20 ↑5.76	43.88 ↑18.99	80.44 ↑2.29	21.82 ↑1.09	57.21 ↑6.48
JoyAgent	61.33 ↑7.56	70.80 ↑0.40	62.34 ↑5.90	43.22 ↑18.33	79.80 ↑1.65	21.20 ↑0.47	56.45 ↑5.72
OAgents	60.63 ↑6.86	71.30 ↑0.90	61.79 ↑5.35	41.90 ↑17.01	78.86 ↑0.71	22.83 ↑2.10	56.22 ↑5.49
LatentMem	65.98 ↑12.21	78.90 ↑8.50	64.46 ↑8.02	44.14 ↑19.25	81.49 ↑3.34	25.13 ↑4.40	60.02 ↑9.29
Held-out · CAMEL
No-memory	56.96	70.70	58.04	32.38	79.14	22.10	53.22
ChatDev	57.55 ↑0.59	68.20 ↓2.50	59.04 ↑1.00	36.78 ↑4.40	78.12 ↓1.02	18.58 ↓3.52	53.04 ↓0.18
MetaGPT	59.06 ↑2.10	69.90 ↓0.80	59.38 ↑1.34	37.47 ↑5.09	80.53 ↑1.39	22.55 ↑0.45	54.82 ↑1.60
Generative	57.63 ↑0.67	70.65 ↓0.05	61.20 ↑3.16	35.32 ↑2.94	80.35 ↑1.21	17.29 ↓4.81	53.74 ↑0.52
Voyager	56.57 ↓0.39	69.85 ↓0.85	58.27 ↑0.23	36.25 ↑3.87	79.91 ↑0.77	23.65 ↑1.55	54.08 ↑0.86
G-Memory	59.20 ↑2.24	70.40 ↓0.30	60.12 ↑2.08	38.04 ↑5.66	79.28 ↑0.14	24.56 ↑2.46	55.27 ↑2.05
JoyAgent	58.10 ↑1.14	70.20 ↓0.50	61.34 ↑3.30	37.50 ↑5.12	79.45 ↑0.31	20.65 ↓1.45	54.54 ↑1.32
OAgents	58.33 ↑1.37	71.40 ↑0.70	60.09 ↑2.05	31.99 ↓0.39	79.28 ↑0.14	13.89 ↓8.21	52.50 ↓0.72
LatentMem	68.74 ↑11.78	77.75 ↑7.05	64.20 ↑6.16	47.23 ↑14.85	80.70 ↑1.56	28.12 ↑6.02	61.12 ↑7.90
Held-out · DyLAN
No-memory	53.87	69.25	57.44	24.89	79.81	23.81	51.51
ChatDev	56.28 ↑2.41	68.55 ↓0.70	60.23 ↑2.79	39.27 ↑14.38	79.82 ↑0.01	16.55 ↓7.26	53.45 ↑1.94
MetaGPT	61.78 ↑7.91	71.50 ↑2.25	62.23 ↑4.79	37.00 ↑12.11	78.25 ↓1.56	19.19 ↓4.62	54.99 ↑3.48
Generative	60.39 ↑6.52	71.70 ↑2.45	60.92 ↑3.48	43.57 ↑18.68	78.86 ↓0.95	18.58 ↓5.23	55.67 ↑4.16
Voyager	59.06 ↑5.19	70.05 ↑0.80	61.79 ↑4.35	40.40 ↑15.51	80.00 ↑0.19	26.37 ↑2.56	56.28 ↑4.77
G-Memory	61.23 ↑7.36	71.35 ↑2.10	63.76 ↑6.32	42.28 ↑17.39	80.35 ↑0.54	17.94 ↓5.87	56.15 ↑4.64
JoyAgent	59.96 ↑6.09	71.35 ↑2.10	62.12 ↑4.68	41.67 ↑16.78	79.89 ↑0.08	25.81 ↑2.00	56.80 ↑5.29
OAgents	60.98 ↑7.11	71.15 ↑1.90	61.35 ↑3.91	42.36 ↑17.47	78.77 ↓1.04	19.75 ↓4.06	55.73 ↑4.21
LatentMem	65.55 ↑11.68	78.80 ↑9.55	65.22 ↑7.78	44.25 ↑19.36	81.40 ↑1.59	29.08 ↑5.27	60.72 ↑9.21

5. Experiments

5.1 Experimental Setup

Datasets and Benchmarks. Our evaluation covers six benchmarks across four domains: (1) Knowledge-intensive QA: TriviaQA and PopQA; (2) Coding: KodCode and BigCodeBench; (3) Reasoning QA: StrategyQA; and (4) Symbolic Planning: PDDL. Detailed information for these benchmarks are provided in the appendix.

数据集与基准。 我们的评估覆盖四个领域的六个基准：(1) 知识密集型问答：TriviaQA 和 PopQA；(2) 编程：KodCode 和 BigCodeBench；(3) 推理问答：StrategyQA；以及 (4) 符号规划：PDDL。 这些基准的详细信息见附录。

Baselines. Besides memory-free methods, we select three representative single-agent memory baselines, including Voyager, Generative, and JoyAgent, as well as four multi-agent memory baselines adopted from mainstream MAS frameworks: MetaGPT, ChatDev, OAgents, JoyAgent, and MAS-specific G-Memory. Additional details are provided in the appendix.

基线。 除无记忆方法外，我们选择了三种代表性的单智能体记忆基线，包括 Voyager、Generative 和 JoyAgent，以及来自主流 MAS 框架的四种多智能体记忆基线：MetaGPT、ChatDev、OAgents、JoyAgent 和 MAS 专用的 G-Memory。 更多细节见附录。

MAS and LLM Backbones. Four representative multi-agent frameworks are adopted to integrate with LatentMem and the baselines, including AutoGen, MacNet, CAMEL and DyLAN. More details on the MAS setups are placed in the appendix. To instantiate these MAS frameworks, we adopt two common LLMs with different sizes, i.e., Qwen/Qwen3-4B-Instruct-2507 and meta-llama/Llama-3.1-8B-Instruct.

MAS 与 LLM 骨干。 我们采用四种代表性的多智能体框架来整合 LatentMem 和各基线，包括 AutoGen、MacNet、CAMEL 和 DyLAN。 MAS 设置的更多细节见附录。 为实例化这些 MAS 框架，我们采用两个不同规模的常用 LLM，即 Qwen/Qwen3-4B-Instruct-2507 和 meta-llama/Llama-3.1-8B-Instruct。

Training Configurations. We implement the embedding function $\mathbf{v}(\cdot )$ mentioned above with the all-MiniLM-L6-v2 model. The memory composer $\mathcal{C}$ is realized as a lightweight transformer, with its parameters initialized from the backbone LLM and trained using LoRA. We set $K=1$ in the retrieval equation and fix the latent memory sequence length to $L^{\prime }=8$. The ablation study on hyper-parameter settings is reported in Section 5.6. Detailed training setups and parameter configurations are listed in the appendix.

训练配置。 我们使用 all-MiniLM-L6-v2 模型实现前述嵌入函数 $\mathbf{v}(\cdot )$。 记忆合成器 $\mathcal{C}$ 被实现为轻量级 Transformer，其参数由骨干 LLM 初始化，并使用 LoRA 训练。 我们在检索公式中设置 $K=1$，并把潜在记忆序列长度固定为 $L^{\prime }=8$。 超参数设置的消融研究见第 5.6 节。 详细训练设置和参数配置见附录。

5.2 Main Results

LatentMem Delivers High-Performance Memory Across Domains and MAS Frameworks. As shown in Table 1, when integrated with in-domain MAS frameworks such as AutoGen and MacNet that are powered by Qwen3-4B-Instruct-2507, LatentMem outperforms the state-of-the-art single- and multi-agent memory baselines by an average of $7.86\mathrm{\%}$ and $6.66\mathrm{\%}$, respectively. Notably, it yields a $16.20\mathrm{\%}$ improvement for AutoGen on the TriviaQA benchmark. Furthermore, LatentMem exhibits strong scalability with the model size increases. As shown in the appendix, it elevates MacNet’s performance on KodCode from $48.50\mathrm{\%}$ to $65.50\mathrm{\%}$ using Llama-3.1-8B-Instruct.

LatentMem 在不同领域与 MAS 框架上提供高性能记忆。 如表1所示，当 LatentMem 与由 Qwen3-4B-Instruct-2507 驱动的 AutoGen、MacNet 等域内 MAS 框架结合时，它平均分别比最先进的单智能体和多智能体记忆基线高出 $7.86\mathrm{\%}$ 和 $6.66\mathrm{\%}$。 值得注意的是，它在 TriviaQA 基准上使 AutoGen 提升了 $16.20\mathrm{\%}$。 此外，随着模型规模增加，LatentMem 表现出很强的可扩展性。 如附录所示，使用 Llama-3.1-8B-Instruct 时，它把 MacNet 在 KodCode 上的性能从 $48.50\mathrm{\%}$ 提高到 $65.50\mathrm{\%}$。

LatentMem Exhibits Strong Generalization Capability. On out-of-domain benchmarks, most MAS memory methods fail to generalize. As shown in Table 1, LatentMem improves AutoGen on PDDL by $7.10\mathrm{\%}$, while MetaGPT and Voyager drop by up to $4.44\mathrm{\%}$ and $2.77\mathrm{\%}$, respectively. Similarly, on previously unseen MAS frameworks, LatentMem boosts CAMEL on KodCode by $7.05\mathrm{\%}$, whereas nearly all baselines decline. We attribute these gaps to the rigid and homogeneous memory designs of existing methods, which limit adaptability and representational capacity. These results demonstrate LatentMem’s robustness across domains, agent roles, and collaboration patterns, highlighting the importance of role-aware memory for generalizable MAS.

LatentMem 展现出强泛化能力。 在域外基准上，大多数 MAS 记忆方法无法实现良好泛化。 如表1所示，LatentMem 使 AutoGen 在 PDDL 上提升 $7.10\mathrm{\%}$，而 MetaGPT 和 Voyager 分别最多下降 $4.44\mathrm{\%}$ 和 $2.77\mathrm{\%}$。 类似地，在此前未见的 MAS 框架上，LatentMem 使 CAMEL 在 KodCode 上提升 $7.05\mathrm{\%}$，而几乎所有基线都有所下降。 我们把这些差距归因于现有方法僵化且同质的记忆设计，它们限制了适应性和表示能力。 这些结果表明 LatentMem 对不同领域、智能体角色和协作模式均具有鲁棒性，并凸显了角色感知记忆对于可泛化 MAS 的重要性。

5.3 Cost Analysis

图3：LatentMem 的时间与 token 消耗。每个面板展示不同记忆架构下性能与资源成本之间的权衡：上排绘制性能与时间，下排绘制性能与 token 成本；圆的面积反映相对资源消耗。

As shown in Figure 3, LatentMem achieves the largest performance gains among memory-based baselines while incurring minimal time and token costs. It delivers the greatest improvement on TriviaQA for DyLAN ($+11.68\mathrm{\%}$ over No-memory) with substantially lower time overhead (e.g., cutting inference time by a factor of $2.16\times$ relative to OAgents), and achieves the highest gain on KodCode for AutoGen ($+8.40\mathrm{\%}$) while using even fewer tokens than No-Memory (0.01M tokens less). In contrast, JoyAgent consumes $1.87$M additional tokens for only a $2.50\mathrm{\%}$ gain, highlighting the superior efficiency of LatentMem.

如图3所示，LatentMem 在记忆基线中取得最大的性能提升，同时只产生很少的时间与 token 成本。 它在 DyLAN 的 TriviaQA 上取得最大提升（相比 No-memory 提升 $+11.68\mathrm{\%}$），且时间开销显著更低（例如推理时间较 OAgents 缩短至 $1/2.16$）；它还在 AutoGen 的 KodCode 上取得最高增益（$+8.40\mathrm{\%}$），使用的 token 甚至比 No-Memory 更少（少 0.01M token）。 相比之下，JoyAgent 额外消耗 $1.87$M token，却只取得 $2.50\mathrm{\%}$ 的提升，凸显了 LatentMem 更优的效率。

5.4 Comparison with Multi-Agent Fine-Tuning

To assess the effectiveness of LatentMem under comparable training conditions, we compare it with a representative multi-agent fine-tuning baseline, MARTI. MARTI is trained with GRPO under exactly the same computational budget as LatentMem, where all agents share a single LLM backbone and are trained on the same datasets (TriviaQA, KodCode, StrategyQA, and PopQA), ensuring a fully fair and controlled comparison.

为评估 LatentMem 在可比训练条件下的有效性，我们将其与代表性的多智能体微调基线 MARTI 进行比较。 MARTI 与 LatentMem 在完全相同的计算预算下使用 GRPO 训练，所有智能体共享单一 LLM 骨干，并在相同数据集（TriviaQA、KodCode、StrategyQA 和 PopQA）上训练，从而确保完全公平且受控的比较。

表2：在 AutoGen 和 MacNet 两种 MAS 框架上，比较多智能体微调方法 MARTI 与 LatentMem 在 KodCode 和 TriviaQA 上的性能。

MAS	Method	KodCode	TriviaQA
AutoGen	MARTI	74.20	64.78
	LatentMem	76.80 ↑2.60	76.51 ↑11.73
MacNet	MARTI	73.10	62.31
	LatentMem	78.90 ↑5.80	65.98 ↑3.67

As shown in Table 2, LatentMem consistently outperforms direct agent backbone fine-tuning across all settings. Notably, on the TriviaQA dataset with the AutoGen framework, LatentMem achieves a substantial improvement of $11.73\mathrm{\%}$. Moreover, on more complex MAS settings such as MacNet, MARTI experiences a $1.10\mathrm{\%}$ performance drop on KodCode compared to AutoGen, whereas LatentMem instead surpasses its AutoGen counterpart by $2.10\mathrm{\%}$. These results indicate that LatentMem better exploits the structural advantages of complex MAS, leading to stronger performance gains than direct backbone fine-tuning.

如表2所示，LatentMem 在所有设置中都持续优于直接微调智能体骨干的方法。 值得注意的是，在 AutoGen 框架的 TriviaQA 数据集上，LatentMem 取得了 $11.73\mathrm{\%}$ 的显著提升。 此外，在 MacNet 等更复杂的 MAS 设置中，MARTI 在 KodCode 上相比 AutoGen 性能下降 $1.10\mathrm{\%}$，而 LatentMem 反而比其 AutoGen 对应设置高出 $2.10\mathrm{\%}$。 这些结果表明，LatentMem 能够更好地利用复杂 MAS 的结构优势，因而比直接微调骨干取得更强的性能增益。

5.5 Framework Analysis

图4：LatentMem 在不同数据集与 MAS 框架上生成的潜在记忆的 t-SNE 可视化。

LatentMem Consistently Delivers Role-Aware Memory. As shown in Figure 4, LatentMem consistently generates role-specific latent memories across both in-domain and out-of-domain datasets, as well as seen and unseen MAS. In the left panel (in-domain KodCode, seen MAS AutoGen), user-proxy and assistant memories form two clearly separated clusters. In the right panel (out-of-domain BigCodeBench, unseen MAS CAMEL), the role-specific memories remain well separated, demonstrating LatentMem’s ability to avoid homogeneous memory even in entirely novel task domains, agent roles, and collaboration patterns.

LatentMem 持续提供角色感知记忆。 如图4所示，LatentMem 在域内和域外数据集，以及已见和未见 MAS 上，都能持续生成角色特定的潜在记忆。 在左侧面板中（域内 KodCode、已见 MAS AutoGen），用户代理和助手的记忆形成两个清晰分离的簇。 在右侧面板中（域外 BigCodeBench、未见 MAS CAMEL），角色特定记忆仍然良好分离，表明即使面对全新的任务领域、智能体角色和协作模式，LatentMem 也能避免记忆同质化。

图5：累计准确率（奖励）随问题索引的演化。索引 $i$ 处的累计准确率定义为前 $i$ 个问题的平均准确率（奖励）。

LatentMem Scales Efficiently as Task Horizon Expands. We visualize the cumulative gains of different memory systems as tasks progress, specifically by tracking their impact on cumulative accuracy. As shown in Figure 5, LatentMem steadily improves as more experiences are collected, surpassing all baselines that rely on complex, multi-granularity memory. Although early performance exhibits higher variance due to limited samples, LatentMem quickly stabilizes and continues to improve, demonstrating its ability to efficiently distill high-utility, transferable knowledge from past interaction trajectories, which can then be leveraged to guide the reasoning process of MAS.

随着任务长度扩展，LatentMem 能够高效扩展。 我们通过跟踪不同记忆系统对累计准确率的影响，可视化任务进行过程中它们的累计增益。 如图5所示，随着收集到更多经验，LatentMem 稳步改进，并超过所有依赖复杂多粒度记忆的基线。 尽管早期性能因样本有限而表现出更高方差，LatentMem 很快趋于稳定并持续改进，表明它能够从过去的交互轨迹中高效提炼高效用、可迁移的知识，随后利用这些知识指导 MAS 的推理过程。

5.6 Sensitivity & Ablation Study

图6：左：模型性能对潜在记忆长度 $L^{\prime }$ 的敏感性。右：突出记忆合成器和经验库影响的消融结果。

Sensitivity Analysis. We analyze the sensitivity of LatentMem to two key hyperparameters: the latent memory length $L^{\prime }$ and the number of relevant trajectories $K$. As shown in Figure 6 (Left), performance generally improves with larger $L^{\prime }$, but with diminishing returns; balancing accuracy and computational cost, we set $L^{\prime }=8$. The effect of $K$ is detailed in the appendix: while baselines such as G-Memory degrades when $K>3$ due to information overload, LatentMem consistently improves on both TriviaQA and KodCode, demonstrating its ability to distill useful information from redundant trajectories via latent memory.

敏感性分析。 我们分析 LatentMem 对两个关键超参数的敏感性：潜在记忆长度 $L^{\prime }$ 和相关轨迹数量 $K$。 如图6（左）所示，性能通常随 $L^{\prime }$ 增大而提高，但收益逐渐减小；为平衡准确率与计算成本，我们设置 $L^{\prime }=8$。 附录详细给出了 $K$ 的影响：G-Memory 等基线会因信息过载在 $K>3$ 时性能下降，而 LatentMem 在 TriviaQA 和 KodCode 上均持续改进，表明它能够通过潜在记忆从冗余轨迹中提炼有用信息。

Component Ablation. We present ablation studies of LatentMem in Figure 6 (Right), where we introduce two variants: without role and without experience, corresponding to the removal of agent profile guidance ($\gamma$ in the composer equation) and the disabling of real-time updates in the experience bank, respectively. When the memory composer no longer receives agent profiles, resulting in identical latent memories across agents, performance drops slightly for simple MAS such as AutoGen ($2.30\mathrm{\%}$ on KodCode) and more substantially for complex MAS like MacNet ($6.45\mathrm{\%}$), highlighting the importance of agent-aware memory. Disabling real-time updates in the experience bank leads to minor performance degradation on KodCode ($3.60\mathrm{\%}$ on MacNet) but a larger drop on PDDL ($7.63\mathrm{\%}$), demonstrating its crucial role in adapting to complex task distributions. These results underscore the contributions of both components to the overall effectiveness of LatentMem.

组件消融。 我们在图6（右）中给出 LatentMem 的消融研究，并引入两个变体：without role 和 without experience，分别对应移除智能体画像指导（合成器公式中的 $\gamma$）和禁用经验库实时更新。 当记忆合成器不再接收智能体画像，导致不同智能体获得相同潜在记忆时，AutoGen 等简单 MAS 的性能小幅下降（KodCode 上下降 $2.30\mathrm{\%}$），MacNet 等复杂 MAS 的性能下降更为明显（下降 $6.45\mathrm{\%}$），这凸显了智能体感知记忆的重要性。 禁用经验库实时更新会使 KodCode 上的性能小幅下降（MacNet 上下降 $3.60\mathrm{\%}$），但在 PDDL 上下降更大（$7.63\mathrm{\%}$），表明它在适应复杂任务分布方面发挥关键作用。 这些结果凸显了两个组件对 LatentMem 整体有效性的贡献。

5.7 Case Study

图7：LatentMem 案例研究。通过利用角色感知且紧凑的潜在记忆，LatentMem 能够防止步骤重复和盲目遵循检索轨迹等常见 MAS 问题，同时实现角色感知协调与自我纠错。

Figure 7 shows that LatentMem, by providing role-aware memory, can prevent or promptly correct common error patterns in MAS. Vanilla MacNet often suffers from step repetition, while MacNet with OAgents blindly follow the retrieved trajectories, violating task specifications. In contrast, LatentMem’s high-level, role-aware latent memory enables agents to reinforce role compliance and coordinate effectively, allowing the MAS to self-correct short-term errors and complete tasks successfully. Detailed trajectories and error analyses are in the appendix.

图7表明，通过提供角色感知记忆，LatentMem 能够防止或及时纠正 MAS 中的常见错误模式。 无记忆的 MacNet 经常出现步骤重复，而结合 OAgents 的 MacNet 会盲目遵循检索到的轨迹，违反任务规范。 相比之下，LatentMem 的高层角色感知潜在记忆使智能体能够强化角色遵循并有效协调，让 MAS 可以自行纠正短期错误并成功完成任务。 详细轨迹和错误分析见附录。

6. Conclusion

In this work, we present LatentMem, a latent memory framework for multi-agent systems that enables role-aware and token-efficient memory customization. By leveraging a lightweight experience bank and a learnable memory composer, each agent receives its latent memories distilled from raw trajectories, naturally reinforcing role compliance and enhancing coordination. We further introduce Latent Memory Policy Optimization, which encourages the composer to produce transferable, high-utility latent representations, enhancing generalization across diverse task domains and MAS frameworks. Extensive experiments on six benchmarks and four MAS frameworks demonstrate that LatentMem achieves substantial performance gains, robust generalization, and high efficiency, while effectively mitigating memory homogenization and information overload.

在这项工作中，我们提出 LatentMem，这是一种面向多智能体系统的潜在记忆框架，支持角色感知且节省 token 的记忆定制。 通过利用轻量级经验库和可学习记忆合成器，每个智能体都能获得从原始轨迹中提炼出的潜在记忆，自然强化角色遵循并增强协调。 我们进一步提出潜在记忆策略优化，促使合成器生成可迁移、高效用的潜在表示，增强其在不同任务领域和 MAS 框架上的泛化能力。 在六个基准和四种 MAS 框架上的大量实验表明，LatentMem 在有效缓解记忆同质化和信息过载的同时，实现了显著性能增益、稳健泛化和高效率。

Impact Statement

Ethical Considerations. This study focuses on developing and evaluating multi-agent memory mechanisms using publicly accessible benchmarks and datasets. It does not involve the collection, processing, or deployment of private, personal, or sensitive user information, and all experiments are carried out in controlled, offline research environments. Consequently, we do not foresee any major ethical risks associated with this work.

伦理考量。 本研究聚焦于使用公开可访问的基准和数据集开发与评估多智能体记忆机制。 它不涉及收集、处理或部署私有、个人或敏感用户信息，所有实验均在受控的离线研究环境中开展。 因此，我们认为这项工作不会带来任何重大伦理风险。

Societal Implications. The methods proposed in this paper seek to enhance the robustness and reliability of LLM-based multi-agent systems, with potential benefits for applications including assistive robotics, information organization, and long-term decision-making. However, more powerful memory capabilities may also increase the potential for misuse if such systems are deployed without proper safeguards. As such, we position this work primarily as a research contribution and stress that real-world deployment should be accompanied by appropriate oversight, safety assessments, and compliance with legal and ethical standards.

社会影响。 本文提出的方法旨在增强基于 LLM 的多智能体系统的鲁棒性和可靠性，并可能有益于辅助机器人、信息组织和长期决策等应用。 然而，如果在缺乏适当保障措施的情况下部署此类系统，更强的记忆能力也可能增加被滥用的风险。 因此，我们主要把这项工作定位为研究贡献，并强调真实世界部署应配备适当监督和安全评估，同时遵守法律与伦理标准。