---

Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning

MemoryAgentRLEMNLP 2026中科院计算所中国科学院大学理想汽车

Zhang Kehao, Gui Shangtong, Yang Sheng, et al. Learning to remember: end-to-end training of memory agents for long-context reasoning[PP/OL]. V2. arXiv (2026-09-01)[2026-10-23]. DOI:10.48550/arXiv.2602.18493.

https://arxiv.org/abs/2602.18493v2

https://github.com/ictnlp/unified-memory-agent

---

学会记忆：面向长上下文推理的记忆智能体端到端训练

Abstract

Long-context LLMs and Retrieval-Augmented Generation defer state tracking and evidence consolidation to query time, which is brittle when facts evolve and answers depend on latent states. We introduce Unified Memory Agent (UMA) for a one-to-many setting: query-agnostic external memory is constructed once from a stream and reused across multiple future QA sessions. A single policy maintains a structured Memory Bank through CRUD operations and answers using both the Memory Bank and raw context. Task-Stratified GRPO uses the mean reward of QA trajectories branching from each sampled memory state to supervise memory maintenance, while normalizing memory and per-question QA groups separately.

长上下文 LLM 与检索增强生成将状态跟踪和证据整合推迟到查询时进行；当事实不断变化、答案依赖潜在状态时，这种方式就显得脆弱。 我们面向一对多场景提出统一记忆智能体（Unified Memory Agent，UMA）：从数据流中一次性构建与查询无关的外部记忆，并在未来的多个问答会话中复用。 单一策略通过增删改查（CRUD）操作维护结构化记忆库（Memory Bank），并结合记忆库与原始上下文作答。 任务分层 GRPO 使用从每个采样记忆状态分支得到的问答轨迹的平均奖励来监督记忆维护，同时分别对记忆组和各个问题的问答组进行归一化。

We also introduce Ledger-QA, a diagnostic benchmark for long-horizon state tracking over accumulated updates. At the 16k budget, UMA-Generalist achieves the highest average score among compared methods across the test-time-learning and accurate-retrieval benchmarks and transfers to Ledger-QA without task-specific training; UMA-Specialist further improves long-horizon tracking after task adaptation. These results support learned proactive memory management for long-context reasoning.

我们还提出 Ledger-QA，这是一个基于累积更新进行长时程状态跟踪的诊断基准。 在 16k 预算下，UMA-Generalist 在测试时学习和精确检索基准上的平均得分高于所有对比方法，并且无需针对该任务训练即可迁移到 Ledger-QA；UMA-Specialist 则在任务适配后进一步改善了长时程跟踪表现。 这些结果支持通过学习主动管理记忆，以实现长上下文推理。

1. Introduction

Modern Large Language Models (LLMs) increasingly support contexts of hundreds of thousands of tokens, with some reaching million-token windows. Yet long-context modeling faces challenges beyond window size: standard attention scales quadratically with sequence length, making very long inputs costly, while longer inputs can exhibit lost-in-the-middle effects and broader context rot even when relevant information is present.

现代大语言模型（LLM）越来越多地支持数十万 token 的上下文，其中一些模型的窗口已达到百万 token。 然而，长上下文建模面临的挑战不止于窗口大小：标准注意力的开销随序列长度呈平方增长，使超长输入的处理成本高昂；同时，即使相关信息已经存在，更长的输入仍可能出现“中间信息遗失”效应以及更广泛的 上下文退化（context rot）。

Retrieval-Augmented Generation (RAG) scales by retrieving relevant passages at query time, but shares the same passive information processing limitation with long-context approaches: context is treated as static raw material and reasoning is deferred until a query arrives. As Figure 1 shows, RAG reprocesses retrieved passages for every query; on state tracking, contradiction resolution, and evidence aggregation, this creates both accuracy degradation and computational redundancy.

检索增强生成（RAG）通过在查询时检索相关段落来实现扩展，但它与长上下文方法具有相同的 被动信息处理 局限：上下文被视为静态原始材料，推理则推迟到查询到来后才进行。 如 图1 所示，RAG 对每个查询都要重新处理检索到的段落；在状态跟踪、矛盾消解和证据聚合任务中，这既会降低准确率，也会造成计算冗余。

Memory-augmented agents instead consolidate information as it arrives. Recent work explores RL-trained memory operations, downstream QA rewards, persistent structured stores, query-conditioned end-to-end memory-to-answer policies, and query-hidden memory construction. These advances expose a distinct optimization challenge: how to jointly train a reusable memory state against multiple future questions. In this one-to-many setting, memory is constructed before the questions are revealed and subsequently shared by multiple independent QA sessions. The shared state must preserve broadly useful evidence rather than optimize for a single revealed objective, making supervision across questions central to training. Query-conditioned agents instead construct memory for a known question, whereas query-agnostic systems generally separate memory and QA optimization or associate each memory rollout with a single downstream task.

记忆增强智能体 则在信息到达时就对其进行整合。 近期研究探索了经强化学习训练的记忆操作、下游问答奖励、持久化结构存储、以查询为条件的端到端记忆到答案策略，以及在查询不可见时构建记忆的方法。 这些进展揭示了一项独特的优化挑战：如何针对未来的多个问题，联合训练一个可复用的记忆状态。 在这种 一对多 场景中，记忆在问题揭示之前构建，随后由多个独立的问答会话共享。 共享状态必须保留具有广泛用途的证据，而不是针对单个已经揭示的目标进行优化，因此跨问题监督成为训练的核心。 相比之下，以查询为条件的智能体针对一个已知问题构建记忆，而与查询无关的系统通常将记忆优化和问答优化分开，或将每次记忆 rollout 与单一下游任务关联。

We propose Unified Memory Agent (UMA) to address this optimization problem. For each context, UMA samples memory-maintenance trajectories and branches each resulting memory state into multiple QA trajectories. Their mean utility provides a cross-stage, session-level learning signal for upstream memory maintenance, while Task-Stratified GRPO normalizes memory rollouts and each question's QA rollouts separately. This directly connects query-agnostic memory construction to the downstream utility of the shared state.

我们提出 统一记忆智能体（UMA） 来解决这一优化问题。 对于每个上下文，UMA 采样多条记忆维护轨迹，并将每个最终记忆状态分支为多条问答轨迹。 这些轨迹的平均效用为上游记忆维护提供跨阶段、会话级的学习信号，而 任务分层 GRPO 分别对记忆 rollout 和每个问题的问答 rollout 进行归一化。 这将与查询无关的记忆构建直接关联到共享状态的下游效用。

We also introduce Ledger-QA, a controlled diagnostic benchmark for continuous state tracking over accumulated updates. We evaluate UMA-Generalist on 13 established test-time-learning and accurate-retrieval benchmarks and on Ledger-QA without task-specific training, and evaluate UMA-Specialist on task-adapted Ledger-QA. At the 16k budget, UMA-Generalist achieves the highest aggregate TTL/AR score among compared methods and transfers to Ledger-QA, while UMA-Specialist further improves long-horizon tracking. Together, the results support proactive memory management as a complement to longer context windows and query-time retrieval.

我们还提出 Ledger-QA，这是一个基于累积更新进行持续状态跟踪的受控诊断基准。 我们在 13 个已有的测试时学习和精确检索基准上评估 UMA-Generalist，并在不进行任务专门训练的情况下于 Ledger-QA 上评估它，同时评估经过任务适配的 UMA-Specialist 在 Ledger-QA 上的表现。 在 16k 预算下，UMA-Generalist 的 TTL/AR 综合得分在对比方法中最高，并能迁移到 Ledger-QA，而 UMA-Specialist 进一步改善了长时程跟踪表现。 总体而言，这些结果支持将主动记忆管理作为更长上下文窗口和查询时检索的补充。

图1：支出跟踪示例：RAG 对每个查询都重新处理日志，而智能体记忆维护结构化状态，并从相关字段中获取信息来作答。

2. Method

图2：UMA 概览。阶段 I 通过对各个文本块执行增删改查操作，增量维护结构化记忆库与核心摘要。阶段 II 同时利用记忆库中的结构化检索和原始上下文检索来回答查询。

2.1. Problem Formulation as MDP

We formulate the long-context reasoning task as a Markov Decision Process (MDP). To handle sequences exceeding the context window, we decompose the input into a stream of chunks $C=\{c_1,\dots ,c_N\}$. The MDP tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$ is defined as follows:

我们将长上下文推理任务形式化为马尔可夫决策过程（MDP）。 为处理超出上下文窗口的序列，我们将输入分解为文本块流 $C=\{c_1,\dots ,c_N\}$。 MDP 元组 $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$ 定义如下：

State Space ($\mathcal{S}$). At step $t$, the state $s_t=({\mathcal{M}}_t,x_t,h_t)$ consists of:

状态空间（$\mathcal{S}$）。 在步骤 $t$，状态 $s_t=({\mathcal{M}}_t,x_t,h_t)$ 包括：

• ${\mathcal{M}}_t=(m_t^{core},{\mathcal{B}}_t)$: The dual-component memory, comprising a high-level summary ($m_t^{core}$) and a structured key-value memory bank (${\mathcal{B}}_t$).
• $x_t$: The current input focus. To circumvent context window limits, this is dynamically switched between a local text chunk $c_k$ (during memory maintenance) and a specific query $q$ (during QA).
• $h_t$: The immediate interaction history, which buffers recent observations and is periodically cleared to manage context length.

• ${\mathcal{M}}_t=(m_t^{core},{\mathcal{B}}_t)$：双组件记忆，由高层摘要（$m_t^{core}$）和结构化键值记忆库（${\mathcal{B}}_t$）组成。
• $x_t$：当前输入焦点。为规避上下文窗口限制，它会在局部文本块 $c_k$（记忆维护阶段）与特定查询 $q$（问答阶段）之间动态切换。
• $h_t$：即时交互历史，用于缓冲近期观测，并会定期清空以控制上下文长度。

Action Space ($\mathcal{A}$). We unify all operations into a single action space, categorized by the phase of interaction. Crucially, read-only tools are available in both phases to ensure the agent can verify beliefs or query the memory at any time. Detailed specifications for executable tools and terminal-action parsing conventions are provided in Appendix A.

动作空间（$\mathcal{A}$）。 我们将所有操作统一到一个动作空间中，并按交互阶段分类。 关键在于，两个阶段都提供只读工具，确保智能体随时能够验证信念或查询记忆。 可执行工具的详细规范及终止动作的解析约定见 附录 A。

• Memory Maintenance Operations (${\mathcal{A}}_{mem}$): Used during the chunk processing phase to maintain the state.

  • Add, Update, Delete: Write operations that create, revise or rename, and remove entries in ${\mathcal{B}}_t$.
  • Retrieve, List: Read-only operations for accessing entries and keys in ${\mathcal{B}}_t$.
  • UpdateCore: A special terminal action that commits the current progress to $m_t^{core}$ and triggers the transition to the next chunk.

• QA Operations (${\mathcal{A}}_{qa}$): Used during the query phase to synthesize the answer.

  • BM25, Embedding: Ranked retrieval actions for lexical and semantic search.
  • Retrieve, List: Read-only operations for accessing entries and keys in ${\mathcal{B}}_t$.
  • Answer: The terminal action to generate the final response.

• 记忆维护操作（${\mathcal{A}}_{mem}$）： 在文本块处理阶段用于维护状态。

  • Add、Update、Delete：写操作，分别用于在 ${\mathcal{B}}_t$ 中创建条目、修订或重命名条目，以及删除条目。
  • Retrieve、List：只读操作，用于访问 ${\mathcal{B}}_t$ 中的条目与键。
  • UpdateCore：一种特殊的终止动作，将当前进展提交到 $m_t^{core}$，并触发向下一个文本块的转移。

• 问答操作（${\mathcal{A}}_{qa}$）： 在查询阶段用于综合生成答案。

  • BM25、Embedding：排序检索动作，分别用于词法搜索与语义搜索。
  • Retrieve、List：只读操作，用于访问 ${\mathcal{B}}_t$ 中的条目与键。
  • Answer：生成最终回复的终止动作。

Transition Dynamics ($\mathcal{P}$). The system evolves through two nested loops plus a final phase switch:

转移动态（$\mathcal{P}$）。 系统通过两个嵌套循环以及最终的阶段切换来演进：

• Inner Loop (within one session/chunk): The agent performs multiple tool calls before committing the chunk. Each step appends interaction traces (tool calls and observations) to $h_t$ and may update the memory bank ${\mathcal{B}}_t$, while the current input focus $x_t=c_k$ remains unchanged.
• Outer Loop (across chunks): A terminal UpdateCore action ends the current inner-loop episode, writes the consolidated update to $m_t^{core}$, advances $x_t$ from $c_k$ to $c_{k+1}$, and re-initializes $h_t$ for the next chunk.
• Phase Transition: After processing $c_N$, the system switches from maintenance to QA, where $x_t$ becomes a query $q$ sampled from $P(q|C)$.

• 内循环（单个会话／文本块内）： 智能体在提交文本块之前执行多次工具调用。每一步都将交互轨迹（工具调用与观测）追加到 $h_t$，并可能更新记忆库 ${\mathcal{B}}_t$，而当前输入焦点 $x_t=c_k$ 保持不变。
• 外循环（跨文本块）： 终止动作 UpdateCore 结束当前内循环回合，将整合后的更新写入 $m_t^{core}$，把 $x_t$ 从 $c_k$ 推进到 $c_{k+1}$，并为下一个文本块重新初始化 $h_t$。
• 阶段转移： 处理完 $c_N$ 后，系统从维护阶段切换到问答阶段，此时 $x_t$ 变为从 $P(q|C)$ 中采样的查询 $q$。

This transition structure implies that the policy should both optimize short-horizon inner-loop edits on ${\mathcal{B}}_t$ and optimize outer-loop consolidation into $m_t^{core}$, so that the final memory state ${\mathcal{M}}_N$ supports future queries.

这种转移结构意味着，策略既应优化对 ${\mathcal{B}}_t$ 的短时程内循环编辑，也应优化外循环中向 $m_t^{core}$ 的整合，使最终记忆状态 ${\mathcal{M}}_N$ 能够支持未来查询。

2.2. Unified Memory Agent Architecture

Based on the MDP formulation, UMA employs a unified policy ${\pi }_{\theta }(a_t|s_t)$ to generate actions. The architecture is designed to handle the inputs explicitly as follows:

基于上述 MDP 形式化，UMA 使用统一策略 ${\pi }_{\theta }(a_t|s_t)$ 生成动作。 该架构对输入的具体处理方式如下：

Input Representation. At any time step $t$, the input context provided to the LLM is a concatenation of four segments:

输入表示。 在任意时间步 $t$，提供给 LLM 的输入上下文由以下四部分拼接而成：

$$\begin{array}{}\text{(1)}& {\text{Input}}_t=[I_{sys},m_t^{core},x_t,h_t]\end{array}$$

• System Instruction ($I_{sys}$): Specifies the phase-specific operational guidelines—governing either memory maintenance or question answering—and defines the available toolset (${\mathcal{A}}_{mem}$ or ${\mathcal{A}}_{qa}$).
• Core Memory ($m_t^{core}$): The evolving high-level summary that provides global context.
• Current Focus ($x_t$): The raw text of the current chunk $c_k$ (in Phase I) or the user query $q$ (in Phase II).
• Interaction History ($h_t$): The trajectory of recent tool calls and their execution results.

• 系统指令（$I_{sys}$）： 指定各阶段的操作准则，分别约束记忆维护或问答，并定义可用工具集（${\mathcal{A}}_{mem}$ 或 ${\mathcal{A}}_{qa}$）。
• 核心记忆（$m_t^{core}$）： 持续演化的高层摘要，用于提供全局上下文。
• 当前焦点（$x_t$）： 当前文本块 $c_k$ 的原始文本（阶段 I）或用户查询 $q$（阶段 II）。
• 交互历史（$h_t$）： 近期工具调用及其执行结果的轨迹。

The Unified Memory Agent framework contains two phases, as illustrated in Figure 2.

统一记忆智能体框架包含两个阶段，如 图2 所示。

Phase I: Sequential Memory Maintenance. For each chunk $c_k$, the agent initializes $h_t=\mathrm{\varnothing }$. It iteratively samples actions

阶段 I：顺序记忆维护。 对每个文本块 $c_k$，智能体初始化 $h_t=\mathrm{\varnothing }$。 它迭代采样动作

$$\begin{array}{}\text{(2)}& a_t\sim {\pi }_{\theta }(\cdot |I_{sys},m_t^{core},c_k,h_t)\end{array}$$

The environment executes the tool and appends the result to $h_t$. This loop continues until the agent outputs UpdateCore, which compresses relevant insights into $m_t^{core}$ and advances the process to $c_{k+1}$.

环境执行工具，并将结果追加到 $h_t$。 这一循环持续进行，直到智能体输出 UpdateCore，将相关见解压缩到 $m_t^{core}$ 中，并把流程推进到 $c_{k+1}$。

Phase II: Hybrid Retrieval-Augmented QA. After processing the final chunk, the input focus $x_t$ switches to the query $q$. The agent utilizes the fully constructed memory ${\mathcal{M}}_N$ and an empty history $h_t$. It can invoke hybrid retrieval tools (exact-title access via Retrieve and ranked search via BM25/Embedding) to gather evidence. The process terminates when the agent generates the Answer action.

阶段 II：混合检索增强问答。 处理完最后一个文本块后，输入焦点 $x_t$ 切换为查询 $q$。 智能体使用已经完整构建的记忆 ${\mathcal{M}}_N$ 和空历史 $h_t$。 它可以调用混合检索工具来收集证据，包括通过 Retrieve 按精确标题访问，以及通过 BM25/Embedding 进行排序检索。 当智能体生成 Answer 动作时，流程终止。

2.3. Training: Task-Stratified GRPO

图3：任务分层 GRPO 示意图。对于给定输入，首先采样多条记忆轨迹（蓝色），每条轨迹产生一个最终记忆状态；然后针对共享问题集，将每个状态分支为多个问答会话（红色）。右侧：优势在不同组内归一化；记忆会话共享一个记忆组 Gₘₑₘ，而问答会话按具体问题分组为 Gqa,j。底部：奖励结合即时工具执行反馈 rtool 与结果评估 routcome。每个记忆会话接收其所属轨迹的下游问答平均结果，以及自身的工具奖励。

We propose a Task-Stratified Group Relative Policy Optimization algorithm. By viewing GRPO as a Monte Carlo (MC) advantage estimator without a value network, we adapt it to handle the hierarchical structure of our task as illustrated in Figure 3.

我们提出一种 任务分层组相对策略优化 算法。 将 GRPO 视为一种不依赖价值网络的蒙特卡洛（MC）优势估计器后，我们对其进行适配，使其能够处理本任务的层次结构，如 图3 所示。

2.3.1. Nested Trajectory Sampling

To connect query-agnostic memory construction with downstream QA utility, we employ a nested sampling strategy during training. For a given context $C$:

为将与查询无关的记忆构建关联到下游问答效用，我们在训练中采用嵌套采样策略。 对于给定上下文 $C$：

1. Memory Rollout (trajectory level): We sample $G$ trajectories $\{{\tau }^1,\dots ,{\tau }^G\}$. Each ${\tau }^i$ comprises $N$ memory-tool-calling sessions $\{s_{mem}^{i,1},\dots ,s_{mem}^{i,N}\}$, one per context chunk, and yields a final state ${\mathcal{M}}_N^i$.
2. Shared Query Sampling: Once per context, we sample ${\mathcal{Q}}_C=\{q_j{\}}_{j=1}^M$ and reuse it across all ${\mathcal{M}}_N^i$. Here, $M$ denotes the number of questions for the current training instance and varies across datasets.
3. QA Rollout (session level): For each pair $({\mathcal{M}}_N^i,q_j)$, we generate one multi-turn QA-tool-calling session $s_{qa}^{i,j}$ and compute its reward $R_{qa}^{i,j}$.

1. 记忆 rollout（轨迹级）： 我们采样 $G$ 条轨迹 $\{{\tau }^1,\dots ,{\tau }^G\}$。每条 ${\tau }^i$ 包含 $N$ 个记忆工具调用会话 $\{s_{mem}^{i,1},\dots ,s_{mem}^{i,N}\}$，每个上下文块对应一个会话，并产生最终状态 ${\mathcal{M}}_N^i$。
2. 共享查询采样： 对每个上下文采样一次 ${\mathcal{Q}}_C=\{q_j{\}}_{j=1}^M$，并在所有 ${\mathcal{M}}_N^i$ 之间复用。这里，$M$ 表示当前训练实例的问题数量，会随数据集而变化。
3. 问答 rollout（会话级）： 对每个二元组 $({\mathcal{M}}_N^i,q_j)$，生成一个多轮问答工具调用会话 $s_{qa}^{i,j}$，并计算其奖励 $R_{qa}^{i,j}$。

Each update therefore contains $G$ memory trajectories and $GM$ QA sessions. We fix $G=16$; complete settings are in Appendix I.

因此，每次更新包含 $G$ 条记忆轨迹和 $GM$ 个问答会话。 我们固定使用 $G=16$；完整设置见 附录 I。

2.3.2. Reward Function

Rewards are computed at the session level, with different outcome assignments for QA and memory sessions. Accordingly, actions within the same tool-calling session share the same outcome-derived advantage.

奖励在 会话 层面计算，问答会话与记忆会话采用不同的结果分配方式。 因此，同一工具调用会话中的动作共享相同的、由结果导出的优势。

For each QA session $s_{qa}^{i,j}$, the reward is

对于每个问答会话 $s_{qa}^{i,j}$，其奖励为

$$\begin{array}{}\text{(3)}& \begin{array}{rl}{R}_{qa}^{i,j}& =\lambda r_{tool}^{qa,i,j}+r_{outcome}^{i,j},\\ r_{outcome}^{i,j}& =r_{outcome}(a_{ans}^{i,j},y_j)\end{array}\end{array}$$

where $r_{tool}^{qa,i,j}$ is the tool success signal of this QA session.

其中，$r_{tool}^{qa,i,j}$ 是该问答会话的工具成功信号。

For each memory session $s_{mem}^{i,k}$ in trajectory ${\tau }^i$, we keep an independent tool reward and use the averaged QA outcome under the same trajectory as its outcome signal:

对于轨迹 ${\tau }^i$ 中的每个记忆会话 $s_{mem}^{i,k}$，我们保留独立的工具奖励，并以同一轨迹下问答结果的均值作为其结果信号：

$$\begin{array}{}\text{(4)}& \begin{array}{rl}{\overline{r}}_{outcome}^i& =\frac{1}{M}\sum _{j=1}^M{r}_{outcome}^{i,j},\\ R_{mem}^{i,k}& =\lambda r_{tool}^{mem,i,k}+{\overline{r}}_{outcome}^i\end{array}\end{array}$$

where $r_{tool}^{mem,i,k}$ is computed from the tool-call validity of memory session $s_{mem}^{i,k}$ (e.g., valid-call ratio $N_{valid}/N_{total}$ for that session). Here, $\lambda$ controls the trade-off between tool-use quality and answer outcome. We use the fixed setting $\lambda =0.5$ across tasks.

其中，$r_{tool}^{mem,i,k}$ 根据记忆会话 $s_{mem}^{i,k}$ 的工具调用有效性计算，例如该会话的有效调用比例 $N_{valid}/N_{total}$。 这里，$\lambda$ 控制工具使用质量与回答结果之间的权衡。 我们在所有任务中均固定采用 $\lambda =0.5$。

2.3.3. Task-Stratified Advantage Estimation

Given the session-level rewards defined above, we compute GRPO advantages via group-wise normalization. Our extension uses two stratified groups for memory and QA, respectively.

给定上文定义的会话级奖励，我们通过组内归一化计算 GRPO 优势。 我们的扩展分别针对记忆与问答采用两个分层组。

Advantage for Memory Sessions ($A_{mem}$). Each memory-session reward is normalized against all memory sessions sampled in the current update:

记忆会话的优势（$A_{mem}$）。 每个记忆会话的奖励都相对于当前更新中采样的全部记忆会话进行归一化：

$$\begin{array}{}\text{(5)}& \begin{array}{rl}{A}_{mem}^{i,k}& =\frac{R_{mem}^{i,k}-{\overline{R}}_{mem}}{\sigma (R_{mem}^{\cdot ,\cdot })+{ϵ}_{std}},\\ {\overline{R}}_{mem}& =\frac{1}{GN}\sum _{u=1}^G\sum _{v=1}^N{R}_{mem}^{u,v}\end{array}\end{array}$$

The resulting advantage is assigned to the session, and all actions within that tool-calling session share it.

所得优势被分配给该会话，该工具调用会话中的所有动作共享这一优势。

Advantage for QA Sessions ($A_{qa}$). For a specific question $q_j$, the difficulty is intrinsic. Therefore, we normalize each QA-session reward only against other sessions answering the same question $q_j$ (but with different constructed memory banks).

问答会话的优势（$A_{qa}$）。 对于特定问题 $q_j$，其难度是内在属性。 因此，我们仅相对于回答 同一 问题 $q_j$ 的其他会话，对每个问答会话的奖励进行归一化；这些会话所用的记忆库则由不同轨迹构建。

$$\begin{array}{}\text{(6)}& \begin{array}{rl}{A}_{qa}^{i,j}& =\frac{R_{qa}^{i,j}-{\overline{R}}_{qa,j}}{\sigma (R_{qa}^{\cdot ,j})+{ϵ}_{std}},\\ {\overline{R}}_{qa,j}& =\frac{1}{G}\sum _{u=1}^G{R}_{qa}^{u,j}\end{array}\end{array}$$

Here, ${ϵ}_{std}={10}^{-6}$ prevents division-by-zero errors. Normalizing within each question group removes cross-question difficulty variation, so advantages reflect relative performance among rollouts for the same question.

这里，${ϵ}_{std}={10}^{-6}$ 用于避免除零错误。 在每个问题组内归一化可消除不同问题之间的难度差异，使优势反映同一问题不同 rollout 之间的相对表现。

Finally, the policy ${\pi }_{\theta }$ is updated by maximizing a unified GRPO objective over all sampled sessions:

最后，通过最大化覆盖所有采样会话的统一 GRPO 目标来更新策略 ${\pi }_{\theta }$：

$$\begin{array}{}\text{(7)}& \mathcal{J}(\theta )=\frac{1}{GN+GM}({\mathcal{S}}_{mem}+{\mathcal{S}}_{qa})\end{array}$$

$$\begin{array}{}\text{(8)}& \begin{array}{rl}{\mathcal{S}}_{mem}& =\sum _{i=1}^G\sum _{k=1}^N\frac{1}{|s_{mem}^{i,k}|}\\ & \cdot \sum _{t\in s_{mem}^{i,k}}\ell ({\rho }_t,A_{mem}^{i,k}),\\ {\mathcal{S}}_{qa}& =\sum _{i=1}^G\sum _{j=1}^M\frac{1}{|s_{qa}^{i,j}|}\\ & \cdot \sum _{t\in s_{qa}^{i,j}}\ell ({\rho }_t,A_{qa}^{i,j})\end{array}\end{array}$$

$$\begin{array}{}\text{(9)}& \begin{array}{rl}\ell ({\rho }_t,A)=& min({\rho }_{t}A,\\ & \text{clip}({\rho }_t,1-ϵ,1+ϵ)A)\\ & -\beta D_{KL,t}\end{array}\end{array}$$

Here, ${\rho }_t=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t\mid s_t)}$ denotes the importance sampling ratio between the current policy ${\pi }_{\theta }$ and the rollout policy ${\pi }_{{\theta }_{\mathrm{old}}}$ used for trajectory collection. In contrast, $D_{KL,t}$ penalizes divergence from the frozen reference policy ${\pi }_{{\theta }_{\mathrm{ref}}}$.

这里，${\rho }_t=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t\mid s_t)}$ 表示当前策略 ${\pi }_{\theta }$ 与用于收集轨迹的 rollout 策略 ${\pi }_{{\theta }_{\mathrm{old}}}$ 之间的重要性采样比率。 相比之下，$D_{KL,t}$ 对偏离冻结参考策略 ${\pi }_{{\theta }_{\mathrm{ref}}}$ 的程度施加惩罚。

$$\begin{array}{rl}{D}_{KL,t}& =D_{KL}({\pi }_{\theta }(\cdot \mid s_t)\parallel {\pi }_{{\theta }_{\mathrm{ref}}}(\cdot \mid s_t))\end{array}$$

3. The Ledger-QA Benchmark

We introduce Ledger-QA, a controllable and extensible synthetic benchmark for long-horizon state tracking. Existing long-term memory benchmarks such as LongMemEval evaluate retrieval and reasoning over extended interaction histories, including temporal reasoning and knowledge updates. Ledger-QA targets a complementary challenge: continuously maintaining an accumulated latent state whose answer need not appear explicitly in any single span. Rather than claiming full realism, it uses simple, transparent generation rules to isolate this capability while remaining challenging for modern long-context agents.

我们提出 Ledger-QA，这是一个用于长时程状态跟踪、可控且可扩展的合成基准。 LongMemEval 等已有长期记忆基准评估在延伸交互历史上的检索与推理能力，包括时间推理和知识更新。 Ledger-QA 关注一项互补的挑战：持续维护累积的潜在状态，而答案不一定在任何单个文本片段中显式出现。 它并不声称完全贴近现实，而是使用简单、透明的生成规则来单独考察这一能力，同时保持对现代长上下文智能体的挑战性。

3.1. Dataset Construction

The construction pipeline contains three stages:

构建流程包含三个阶段：

Timeline & Intent Sampling. We sample $H$ dates from 2023-01-01 to 2024-12-31 and select consumption-scene intents for each session, forming a chronological stream of date-stamped updates.

时间线与意图采样。 我们从 2023-01-01 到 2024-12-31 之间采样 $H$ 个日期，并为每个会话选择消费场景意图，形成按时间排序、带有日期戳的更新流。

Joint Dialogue & Transaction Synthesis. Given each date and its scene intents, a strong LLM (e.g., Gemini 3 Pro) jointly generates natural user-assistant dialogue and aligned structured Transaction records, including context-appropriate amounts. Amounts are not sampled beforehand but determined during this joint synthesis, and the two outputs are returned together to preserve their semantic alignment.

对话与交易联合合成。 给定每个日期及其场景意图，强大的 LLM（例如 Gemini 3 Pro）联合生成自然的用户—助手对话及与之对齐的结构化 Transaction 记录，其中包含符合上下文的金额。 金额并非预先采样，而是在联合合成过程中确定；两类输出一并返回，以保持语义对齐。

Ground-Truth Query Generation. From the resulting structured ledger, we programmatically generate questions and exact answers across 8 categories that require multi-step aggregation over accumulated updates rather than local span matching.

带标准答案的查询生成。 基于得到的结构化账本，我们通过程序生成涵盖 8 个类别的问题及精确答案；这些问题需要对累积更新进行多步聚合，而不是匹配局部文本片段。

A concrete construction example and full generation details are provided in Appendix C.

具体构建示例和完整生成细节见 附录 C。

3.2. Why Ledger-QA is Challenging

Ledger-QA fundamentally differs from standard RAG benchmarks by demanding long-horizon state aggregation rather than local information retrieval. Answers (e.g., "Total spending") are latent values derived from summing scattered transactions ($v_1+\cdots +v_T$) across the timeline. This requires continuous belief-state maintenance: local extraction errors can propagate to final answers, and simple retrieval or approximate summarization is often insufficient. Ledger-QA should therefore be viewed as a controlled diagnostic benchmark rather than a complete proxy for real-world memory use: its simple, transparent rules isolate the prerequisite of consolidating sequential updates into a persistent state. Strong performance does not by itself guarantee robustness in noisier real-world scenarios, but poor performance reveals a weakness in this foundational capability.

Ledger-QA 要求进行 长时程状态聚合，而非局部信息检索，因此在根本上区别于标准 RAG 基准。 答案（例如“总支出”）是通过对时间线上分散的交易求和（$v_1+\cdots +v_T$）得到的潜在值。 这要求持续维护信念状态：局部抽取错误可能传播到最终答案，简单检索或近似摘要通常不足以应对。 因此，应将 Ledger-QA 视为受控诊断基准，而非真实世界记忆使用的完整替代：其简单、透明的规则单独考察了将顺序更新整合为持久状态这一前提能力。 出色的表现本身并不保证在噪声更多的现实场景中具有鲁棒性，但表现不佳则暴露出这一基础能力的薄弱之处。

4. Experiment

4.1. Experimental Setup

4.1.1. Evaluation Benchmarks

We evaluate UMA along three primary axes of memory behavior: Dynamic State Tracking (our Ledger-QA in Section 3), Test-Time Learning (TTL) (TREC-Coarse, TREC-Fine, NLU, Clinic, Banking77, PubMed-RCT), and Accurate Retrieval (AR) (HotpotQA, LoCoMo, LongMemEval, MSC, PerLTQA, SQuAD, ConvoMem).

我们沿记忆行为的三个主要维度评估 UMA：动态状态跟踪（第 3 节 提出的 Ledger-QA）、测试时学习（TTL）（TREC-Coarse、TREC-Fine、NLU、Clinic、Banking77、PubMed-RCT），以及 精确检索（AR）（HotpotQA、LoCoMo、LongMemEval、MSC、PerLTQA、SQuAD、ConvoMem）。

Task focus. Ledger-QA tests persistent state maintenance under continual updates; TTL tests storage and application of task rules from in-context supervision; AR evaluates fine-grained factual recovery from long documents or conversations. Together, they cover distinct memory capabilities rather than repeatedly testing a single aspect.

任务侧重点。 Ledger-QA 测试持续更新下的持久状态维护；TTL 测试从上下文监督中存储并应用任务规则的能力；AR 评估从长文档或对话中恢复细粒度事实的能力。 这些任务共同覆盖不同的记忆能力，而不是反复测试同一个方面。

Evaluation metric. We use an LLM-as-a-Judge protocol with DeepSeek-V3.2 and additionally report deterministic lexical metrics using a shared final-answer extraction and normalization pipeline. Evaluation details, prompt templates, and dataset taxonomies are provided in Appendices K, D, and E.

评估指标。 我们采用基于 DeepSeek-V3.2 的 LLM-as-a-Judge 评估协议，并使用共享的最终答案抽取与归一化流程，额外报告确定性的词汇匹配指标。 评估细节、提示模板和数据集分类体系分别见附录 K、D 和 E。

4.1.2. Baselines

We compare UMA against a diverse set of baselines. First, we evaluate standard approaches: Concat, which directly concatenates the context up to the context budget to test the backbone's native capacity, and RAG, which retrieves the top-20 chunks using a hybrid strategy combining dense embeddings and sparse BM25 via Reciprocal Rank Fusion (RRF). Second, we include recurrent summarization models: MemAgent iteratively updates a running memory state based on the query, while MemAgent-woq performs query-agnostic compression. Finally, we compare against advanced memory frameworks: Mem1 interleaves chunk reading with dynamic retrieval to maintain an evolving internal state, A-MEM is an additional external-memory baseline with explicit memory operations, Mem-α utilizes an RL-optimized policy to manage a composite memory system (core, episodic, and semantic), and Mem-T is a hierarchical memory agent trained in two stages: optimizing retrieval via tree-guided RL and then refining its construction policy through hindsight-based distillation. Implementation details are provided in Appendix F.

我们将 UMA 与多种基线进行比较。 首先评估标准方法：Concat 在上下文预算范围内直接拼接上下文，以测试骨干模型的原生能力；RAG 使用混合策略，通过倒数排名融合（RRF）结合稠密嵌入与稀疏 BM25，检索排名前 20 的文本块。 其次纳入循环摘要模型：MemAgent 根据查询迭代更新持续维护的记忆状态，而 MemAgent-woq 执行与查询无关的压缩。 最后，与先进的记忆框架进行比较：Mem1 将文本块读取与动态检索交错进行，以维护不断演化的内部状态；A-MEM 是另一种具有显式记忆操作的外部记忆基线；Mem-α 使用经强化学习优化的策略来管理复合记忆系统（核心记忆、情景记忆和语义记忆）；Mem-T 则是分两阶段训练的层次化记忆智能体，先通过树引导的强化学习优化检索，再通过基于事后信息的蒸馏改进其构建策略。 实现细节见 附录 F。

4.1.3. Implementation Details

We train two checkpoints with the same architecture and optimization. UMA-Generalist is trained for two epochs on modified HotpotQA and a filtered Mem-α split containing SQuAD, PerLTQA, NLU, TREC-Coarse, PubMed-RCT, and BookSum, with HotpotQA and LongMemEval removed. It is used for TTL/AR and evaluated on Ledger-QA without task-specific training. UMA-Specialist is trained for six epochs on the independently generated Ledger-QA set.

我们使用相同架构和优化方法训练两个检查点。 UMA-Generalist 在修改后的 HotpotQA 和经过筛选的 Mem-α 数据划分上训练两个轮次；后者包含 SQuAD、PerLTQA、NLU、TREC-Coarse、PubMed-RCT 和 BookSum，并移除了 HotpotQA 与 LongMemEval。 它用于 TTL/AR，并在不进行任务专门训练的情况下于 Ledger-QA 上评估。 UMA-Specialist 在独立生成的 Ledger-QA 数据集上训练六个轮次。

Dataset and filtering details are in Appendix H. Dense retrieval uses sentence-transformers/all-MiniLM-L6-v2 served via infinity. UMA builds on customized veRL, initializes from Qwen/Qwen3-4B-Instruct-2507, and trains Task-Stratified GRPO on 32 NVIDIA H200 GPUs. Hyperparameters and default prompts are provided in Appendices I and B.

数据集与筛选细节见 附录 H。 稠密检索使用通过 infinity 提供服务的 sentence-transformers/all-MiniLM-L6-v2。 UMA 基于定制的 veRL 构建，从 Qwen/Qwen3-4B-Instruct-2507 初始化，并在 32 张 NVIDIA H200 GPU 上训练任务分层 GRPO。 超参数与默认提示分别见附录 I 和 B。

4.2. Main Results

表1：UMA-Generalist、消融变体和不同会话预算变体的 DeepSeek-V3.2 LLM-as-a-Judge 得分。粗体表示使用 16k 预算的方法中的最佳结果；下划线表示 UMA-Generalist 的 8k、16k 和 32k 变体中的最佳结果。Pub：PubMed-RCT；Convo：ConvoMem；LoCo：LoCoMo；LME：LongMemEval；PerL：PerLTQA；T-C/T-F：TREC-Coarse/Fine。详见第 4.4 节和原论文附录 K。

Method	Test-Time Learning (TTL)						Accurate Retrieval (AR)							Avg.
	Bank77	Clinic	NLU	Pub	T-C	T-F	Convo	Hotpot	LoCo	LME	MSC	PerL	SQuAD
Baselines
Concat	72.00	79.00	71.00	57.80	71.00	18.00	2.46	63.28	30.01	31.60	58.40	36.00	63.38	50.30
RAG	80.00	70.00	65.00	51.20	9.00	14.00	51.12	78.12	54.03	59.60	54.60	83.75	25.35	53.52
MemAgent	25.00	13.00	22.00	49.90	61.00	37.00	59.60	51.56	60.37	55.40	50.20	65.25	77.02	48.25
MemAgent-woq	27.00	41.00	34.00	44.40	70.00	44.00	1.56	11.72	25.43	7.60	29.40	23.50	25.01	29.59
Mem1	0.00	6.00	0.00	9.00	7.00	0.00	1.34	5.47	21.60	6.40	8.40	7.25	15.09	6.73
A-MEM	83.00	81.00	72.00	53.70	36.00	16.00	52.46	67.97	45.87	31.80	62.80	73.00	88.12	58.75
Mem-α	83.00	77.00	68.00	56.50	78.00	62.00	7.14	79.69	40.85	53.80	56.80	74.50	81.92	63.02
Mem-T	67.00	52.00	58.00	46.80	15.00	7.00	51.12	66.41	41.04	52.40	65.80	56.75	67.03	49.72
Ablations
UMA (w/o RL & Phase I)	26.00	27.00	23.00	39.30	13.00	12.00	79.02	64.06	50.40	38.60	36.20	57.25	82.33	42.17
UMA (w/o Phase I)	84.00	80.00	75.00	47.40	81.00	28.00	86.16	69.53	44.21	48.80	27.40	59.00	83.47	62.61
UMA (w/o RL)	77.00	72.00	61.00	38.20	69.00	20.00	84.82	64.84	44.26	48.60	63.80	51.50	81.19	59.71
UMA (w/o Raw-Context)	69.00	71.00	59.00	36.30	71.00	44.00	73.12	60.15	40.59	35.80	62.20	52.00	76.52	57.74
UMA (Global Group)	69.00	77.00	84.00	72.50	88.00	53.00	81.47	60.94	56.55	37.60	74.40	73.00	81.98	69.96
UMA (Two-Stage)	86.00	84.00	70.00	44.60	77.00	43.00	84.82	67.19	42.75	49.20	65.60	60.75	65.59	64.65
UMA-Generalist (16k)	87.00	91.00	82.00	67.20	94.00	75.00	87.05	76.56	46.58	51.60	66.20	74.50	87.53	75.86
Session-budget scaling
UMA-Generalist (8k)	79.00	88.00	75.00	65.60	95.00	70.00	80.80	64.06	44.91	44.80	64.60	71.25	85.81	71.45
UMA-Generalist (32k)	90.00	92.00	84.00	68.60	97.00	83.00	89.06	74.22	47.48	49.20	67.00	69.00	87.32	76.76

Table 1 presents UMA-Generalist and the baselines across 13 standard benchmarks (6 TTL and 7 AR tasks).

表1 展示了 UMA-Generalist 与各基线在 13 个标准基准（6 个 TTL 任务和 7 个 AR 任务）上的表现。

Results on Test-Time Learning (TTL). UMA-Generalist averages 82.70 across the six TTL tasks, the highest six-task average at the 16k budget, and leads four tasks. Its performance on Banking77 and Clinic, which are unseen during training, also indicates transfer to new classification domains. Performance is also stable across classification formats, suggesting that the learned maintenance policy is not tied to a single label inventory.

测试时学习（TTL）结果。 UMA-Generalist 在六个 TTL 任务上的平均分为 82.70，是 16k 预算下最高的六任务平均分，并在其中四个任务上领先。 它在训练期间未见过的 Banking77 和 Clinic 上的表现，也表明其能够迁移到新的分类领域。 在不同分类形式下，性能也较为稳定，说明学到的维护策略并不局限于某一套标签。

Results on Accurate Retrieval (AR). UMA-Generalist obtains the highest seven-task average of 70.00 and ranks among the top three methods on five tasks. It leads ConvoMem and ranks among the top three methods on HotpotQA, MSC, PerLTQA, and SQuAD. The aggregate gain reflects consistently strong performance across tasks. Efficiency and scalability analysis is provided in Appendix J. These benchmarks reward retaining evidence that remains useful across heterogeneous questions rather than compressing context toward one known target.

精确检索（AR）结果。 UMA-Generalist 取得最高的七任务平均分 70.00，并在五个任务上跻身前三。 它在 ConvoMem 上领先，并在 HotpotQA、MSC、PerLTQA 和 SQuAD 上位列前三。 综合增益反映出其在各任务上一贯较强的表现。 效率与可扩展性分析见 附录 J。 这些基准奖励保留对异质问题持续有用的证据，而不是围绕单个已知目标压缩上下文。

MemAgent-woq and Mem1 are notably weaker on several TTL tasks, where memory is constructed before the target classification query is revealed. This protocol tests whether compact states learned for objective-conditioned use transfer to query-hidden retention.

MemAgent-woq 和 Mem1 在多个 TTL 任务上明显较弱；这些任务要求在目标分类查询揭示之前构建记忆。 这一协议检验的是：为以目标为条件的使用方式学到的紧凑状态，能否迁移到查询不可见时的信息保留场景。

4.3. Results on Ledger-QA

表2：Ledger-QA 平均结果（%）。Judge 使用 DeepSeek-V3.2；LQA Train 表示是否使用 Ledger-QA 训练数据。

Model	LQA Train	Judge	EM	F1	R-L	Sub-EM
UMA (w/o RL)	No	30.45	19.09	20.15	20.15	42.59
MemAgent	No	40.52	19.28	23.63	23.64	42.50
UMA-Generalist (16k)	No	38.18	34.39	36.24	36.24	43.42
UMA-Specialist (16k)	Yes	53.02	49.39	51.62	51.62	55.65

图4：不同会话时程下 Ledger-QA 的性能比较。横轴表示对话会话时程 H，纵轴表示 LLM-as-a-Judge 得分。图中包含参与评估的基线和多个 UMA 变体；完整结果见原论文附录中的表16。

We evaluate Ledger-QA across session horizons $H\in \{2,5,10,20,30,40,50,100,500\}$. Figure 4 compares the baselines and multiple UMA variants at representative horizons. Table 2 reports averages across all horizons and separates transfer from task adaptation; per-horizon Judge and EM results are provided in Appendix M.

我们在会话时程 $H\in \{2,5,10,20,30,40,50,100,500\}$ 下评估 Ledger-QA。 图4 比较了各基线与多个 UMA 变体在代表性时程下的表现。 表2 报告所有时程上的平均结果，并区分迁移与任务适配；各时程的 Judge 和 EM 结果见 附录 M。

Without Ledger-QA training, UMA-Generalist reaches 38.18 LLM-as-a-Judge and 34.39 EM, compared with 30.45 and 19.09 for UMA (w/o RL). MemAgent records a higher Judge score of 40.52 under its query-conditioned protocol, which constructs a separate state after seeing each question. UMA-Generalist reuses one query-hidden memory state across all questions and achieves higher EM, F1, and ROUGE-L. Task adaptation further raises UMA-Specialist to 53.02 Judge and 49.39 EM. The lexical metrics provide a complementary view because they require normalized answer overlap, whereas the Judge can accept semantically correct answers with different surface forms.

在未使用 Ledger-QA 训练数据的情况下，UMA-Generalist 的 LLM-as-a-Judge 得分为 38.18，EM 为 34.39，而 UMA（w/o RL）分别为 30.45 和 19.09。 MemAgent 在其以查询为条件的协议下获得更高的 Judge 得分 40.52；该协议在看到每个问题之后分别构建状态。 UMA-Generalist 对所有问题复用一个在查询不可见时构建的记忆状态，并取得更高的 EM、F1 和 ROUGE-L。 任务适配进一步将 UMA-Specialist 的 Judge 和 EM 分别提升至 53.02 和 49.39。 词汇匹配指标提供了互补视角，因为它们要求归一化后的答案存在重叠，而 Judge 可以接受表面形式不同但语义正确的答案。

Vulnerability of Baselines. RAG and MemAgent-woq remain competitive at $H=2$ but fall to 5.77 and 2.88 at $H=500$; even query-conditioned MemAgent drops to 11.54. These declines show that retrieval and simple summarization struggle to aggregate scattered updates over long histories.

基线的脆弱性。 RAG 和 MemAgent-woq 在 $H=2$ 时仍有竞争力，但在 $H=500$ 时分别降至 5.77 和 2.88；即使以查询为条件的 MemAgent 也降至 11.54。 这些下降表明，检索与简单摘要难以聚合长历史中分散的更新。

Long-Horizon Performance of UMA-Specialist. UMA-Specialist remains more stable, especially for $H\ge 20$. At 500 sessions, it scores 25.00, matching UMA (w/o Raw-Context) and exceeding the strongest non-UMA baseline, MemAgent, at 11.54. This margin supports explicit memory operations for accumulated state at ultra-long horizons.

UMA-Specialist 的长时程表现。 UMA-Specialist 更为稳定，尤其是在 $H\ge 20$ 时。 在 500 个会话下，它获得 25.00 分，与 UMA（w/o Raw-Context）持平，并高于最强非 UMA 基线 MemAgent 的 11.54 分。 这一优势支持使用显式记忆操作来处理超长时程下的累积状态。

Arithmetic Control. We further run a 100,000-example arithmetic-only control (Appendix E) where the model sums five random two-decimal numbers below 1000. Accuracy changes little after UMA-Specialist training (base: 25.41%; UMA-Specialist: 25.14%), suggesting Ledger-QA gains come from memory maintenance and state access rather than improved standalone arithmetic. We provide a manual failure breakdown in Appendix L.

算术对照实验。 我们进一步开展了包含 100,000 个样例的纯算术对照实验（附录 E），要求模型对五个小于 1000、保留两位小数的随机数求和。 UMA-Specialist 训练前后的准确率变化很小（基础模型：25.41%；UMA-Specialist：25.14%），说明 Ledger-QA 上的提升源于记忆维护和状态访问，而非独立算术能力的增强。 我们在 附录 L 中给出了人工错误分类分析。

4.4. Ablation Studies

We ablate architecture and training choices in the lower block of Table 1.

我们在 表1 的下半部分对架构与训练选择进行消融。

Ablation Settings. w/o Phase I removes sequential memory maintenance and relies on QA-time retrieval only. w/o RL keeps the full architecture but removes RL optimization. w/o RL & Phase I removes both mechanisms. w/o Raw-Context disables BM25/Embedding and only allows Memory Bank retrieval at inference.

消融设置。 w/o Phase I 移除顺序记忆维护，仅依赖问答时检索。 w/o RL 保留完整架构，但移除强化学习优化。 w/o RL & Phase I 同时移除上述两种机制。 w/o Raw-Context 禁用 BM25/Embedding，推理时仅允许检索记忆库。

Global Group normalizes all memory and QA sessions in a single global group without task stratification. Two-Stage separately trains memory and QA. 8k/32k Window changes the default 16k session budget covering the initial prompt and later interaction tokens.

Global Group 将所有记忆会话与问答会话放在单个全局组内归一化，不进行任务分层。 Two-Stage 分别训练记忆与问答。 8k/32k Window 改变默认的 16k 会话预算，该预算覆盖初始提示和后续交互 token。

Impact of Memory Maintenance (Phase I). Explicit maintenance improves both untrained and RL models. Before RL, enabling write/update operations outperforms the retrieval-only variant, showing that structured state maintenance already benefits retention. After RL, full UMA exceeds w/o Phase I by 13.3 points on average, confirming that these operations contribute beyond QA-time retrieval.

记忆维护（阶段 I）的影响。 显式维护同时改善了未训练模型与强化学习模型的表现。 在强化学习之前，启用写入／更新操作便优于仅检索变体，说明结构化状态维护本身就有利于信息保留。 经过强化学习后，完整 UMA 的平均分比 w/o Phase I 高 13.3 分，证实这些操作提供了问答时检索之外的贡献。

Impact of RL Training. RL improves both architectures. It strengthens retrieval-only tool use and raises full UMA by 16.2 points on average, with gains from better memory writing, updating, and access. The improvements span both retention-heavy TTL tasks and retrieval-heavy AR tasks instead of concentrating in one benchmark family.

强化学习训练的影响。 强化学习改善了两种架构的表现。 它增强了仅检索架构的工具使用能力，并将完整 UMA 的平均分提高了 16.2 分；增益来自更好的记忆写入、更新和访问。 改进同时覆盖侧重保留的 TTL 任务和侧重检索的 AR 任务，而非集中于某一类基准。

Impact of Stratified Advantage Grouping. The Global Group variant, which normalizes all memory and QA sessions in a single global group, is 5.9 points lower on average. Separate normalization better matches binary QA outcomes and aggregated memory utility.

分层优势分组的影响。 将所有记忆会话与问答会话置于单一全局组中归一化的 Global Group 变体，平均分低了 5.9 分。 分别归一化更适合二元问答结果与聚合后的记忆效用。

Two-Stage vs. Unified End-to-End Training. Two-stage training, which optimizes memory maintenance and QA separately, is 11.2 points lower on average. Joint optimization better links memory construction to downstream QA utility.

两阶段训练与统一端到端训练。 分别优化记忆维护和问答的两阶段训练，平均分低了 11.2 分。 联合优化能够更好地将记忆构建关联到下游问答效用。

Effect of Session Context Budget. Moving from 8k to 16k clearly improves performance, whereas 32k provides only modest additional gains at higher compute and memory cost. We therefore retain 16k as the default trade-off.

会话上下文预算的影响。 从 8k 增加到 16k 可明显改善性能，而 32k 在带来更高计算和内存开销的同时，仅提供有限的额外增益。 因此，我们保留 16k 作为默认折中方案。

Ablation: w/o Raw-Context Retrieval. Disabling raw-context retrieval produces a substantial drop on retrieval-heavy TTL/AR tasks and a small decrease on Ledger-QA. Across 952 Ledger-QA QA traces, UMA-Specialist accesses structured memory in every trace: 878 (92.2%) use only Memory Bank tools, 74 (7.8%) additionally invoke raw-context retrieval, and no trace uses raw-context retrieval alone. This pattern indicates that structured memory is the default access path on Ledger-QA, with raw retrieval used as an occasional supplement.

消融：移除原始上下文检索。 禁用原始上下文检索后，侧重检索的 TTL/AR 任务表现大幅下降，而 Ledger-QA 上仅小幅下降。 在 952 条 Ledger-QA 问答轨迹中，UMA-Specialist 的每条轨迹都访问了结构化记忆：878 条（92.2%）仅使用记忆库工具，74 条（7.8%）还调用了原始上下文检索，没有轨迹仅使用原始上下文检索。 这一模式表明，结构化记忆是 Ledger-QA 上默认的访问途径，原始检索仅作为偶尔使用的补充。

5. Related Work

Internal Memory and Context Compression. Several approaches extend or compress context through internal representations, including parameter-based storage, generative or learned memory tokens and slots, and KV-based compression. Such internal-memory approaches generally require model-level access or modification, whereas external memory systems can remain model-agnostic.

内部记忆与上下文压缩。 多种方法通过内部表示来扩展或压缩上下文，包括基于参数的存储、生成式或可学习的记忆 token 与槽位，以及基于 KV 的压缩。 这类内部记忆方法通常需要模型层面的访问或修改，而外部记忆系统可以保持与模型无关。

Prompt-based External Memory. Prompt-based systems such as MemGPT, Mem0, and SCM maintain external stores through OS-like hierarchies, extracted memories, or graphs. Although black-box compatible, they depend on instruction following, making smaller models unreliable for complex memory updates and retrieval in dynamic settings.

基于提示的外部记忆。 MemGPT、Mem0 和 SCM 等基于提示的系统，通过类似操作系统的层级结构、抽取的记忆或图来维护外部存储。 尽管兼容黑盒模型，它们仍依赖指令遵循，使较小模型在动态场景下执行复杂记忆更新与检索时不够可靠。

Trainable Memory Agents. Recent work replaces hand-designed memory routines with learned policies. MEM1 interleaves retrieval with reasoning. MemAgent and ReMemR1 optimize question-conditioned memory-to-answer trajectories, with the target question guiding memory construction. A-MEM organizes an evolving structured store without policy training. Among query-agnostic learned-memory systems, Memory-R1 trains specialized Memory Manager and Answer Agent policies; Mem-α evaluates one shared memory against multiple questions while optimizing the memory writer through a fixed RAG-based QA pipeline; Mem-T combines hierarchical memory construction and multi-step retrieval with tree-guided rewards; and AgeMem jointly learns long- and short-term memory actions with task execution in a three-stage trajectory.

可训练记忆智能体。 近期研究使用学习得到的策略替代人工设计的记忆流程。 MEM1 将检索与推理交错进行。 MemAgent 和 ReMemR1 优化以问题为条件的记忆到答案轨迹，由目标问题指导记忆构建。 A-MEM 在不训练策略的情况下组织不断演化的结构化存储。 在与查询无关的可学习记忆系统中，Memory-R1 训练专门的记忆管理器和回答智能体策略；Mem-α 使用多个问题评估同一份共享记忆，同时通过固定的 RAG 问答流程优化记忆写入器；Mem-T 结合层次化记忆构建、多步检索与树引导奖励；AgeMem 则在三阶段轨迹中，将长期和短期记忆动作与任务执行联合学习。

Appendix G summarizes these formulations. UMA jointly optimizes one policy over query-agnostic memory construction and multiple downstream QA trajectories, with separate normalization for memory and per-question QA groups.

附录 G 总结了这些形式化方式。 UMA 使用一个策略，对与查询无关的记忆构建和多条下游问答轨迹进行联合优化，并分别对记忆组及每个问题的问答组进行归一化。

6. Conclusion

We propose Unified Memory Agent, which jointly optimizes memory acquisition and utilization via reinforcement learning. Task-Stratified GRPO aligns the objectives of long-horizon memory maintenance and downstream question answering. We also introduce Ledger-QA, a dataset requiring persistent updates to an evolving ledger. At the 16k budget, UMA-Generalist achieves the highest aggregate test-time-learning and accurate-retrieval score among compared methods, and UMA-Specialist improves long-horizon state tracking on Ledger-QA. Ablation studies show the contributions of policy training and explicit memory updates. Together, these results distinguish transfer from task-specific adaptation and support proactive memory management alongside longer contexts and query-time retrieval.

我们提出统一记忆智能体，通过强化学习联合优化记忆获取与利用。 任务分层 GRPO 将长时程记忆维护与下游问答的目标对齐。 我们还提出 Ledger-QA，这是一个要求对不断演化的账本进行持久更新的数据集。 在 16k 预算下，UMA-Generalist 的测试时学习与精确检索综合得分在对比方法中最高，而 UMA-Specialist 改善了 Ledger-QA 上的长时程状态跟踪表现。 消融研究展示了策略训练与显式记忆更新的贡献。 总体而言，这些结果区分了迁移和针对任务的适配，并支持将主动记忆管理与更长上下文及查询时检索共同使用。

Limitations

Several limitations remain. First, we study single-agent settings with a fixed tool interface; collaborative agents and evolving APIs are left for future work. Second, Ledger-QA is a controlled synthetic diagnostic with transparent generation rules, and broader validation on real-world proactive state-maintenance tasks is still needed. Third, the latency and scaling measurements reflect the reported hardware and workloads. Finally, downstream outcomes supervise memory maintenance at the session level, so actions within one tool-calling session share an advantage; identifying the contribution of individual CRUD operations would require finer-grained or counterfactual attribution. Persistent memory also introduces privacy and data-retention risks, discussed below.

仍存在若干局限。 首先，我们研究的是工具接口固定的单智能体场景；协作智能体与持续演化的 API 留待未来研究。 其次，Ledger-QA 是一个生成规则透明的受控合成诊断基准，仍需在真实世界的主动状态维护任务上进行更广泛的验证。 第三，延迟与扩展性测量反映的是文中报告的硬件和工作负载条件。 最后，下游结果在会话层面监督记忆维护，因此同一个工具调用会话中的动作共享优势；若要识别各个 CRUD 操作的贡献，就需要更细粒度或反事实的归因。 持久记忆还会带来隐私和数据留存风险，下文将对此讨论。

Ethical Considerations

Long-term memory agents may retain sensitive personal information across sessions. Deployments should obtain informed consent, minimize stored content, apply retention limits and access controls, and provide users with mechanisms to inspect, correct, and delete memory. Sensitive information should be excluded by default unless its storage is necessary and explicitly authorized. Deletion should cover both memory entries and derived retrieval artifacts, including indexes and caches. Ledger-QA is fully synthetic and contains no real personal or financial records.

长期记忆智能体可能跨会话保留敏感个人信息。 部署时应取得知情同意，尽量减少存储内容，实施留存期限与访问控制，并为用户提供查看、更正和删除记忆的机制。 除非存储确有必要且获得明确授权，否则默认应排除敏感信息。 删除范围应同时覆盖记忆条目和衍生的检索产物，包括索引与缓存。 Ledger-QA 完全由合成数据构成，不包含真实的个人或财务记录。