---

Metis: Memory Foundation Model

MemoryAgent100+中国人民大学新加坡国立大学上海交通大学同济大学

Zhang Z, Guo Z, Sun Y, et al. Metis: Memory Foundation Model[EB/OL]. arXiv:2607.26760, 2026.

https://github.com/MemTensor/Metis

https://huggingface.co/collections/IAAR-Shanghai/metis

---

Metis：记忆基础模型

Abstract

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency.

近年来，AI 智能体不断把原生能力内化到其底层基础模型中，由此催生了多模态基础模型和大型推理模型。 然而，智能体记忆仍主要通过外部模块实现，原生记忆能力尚未得到充分探索。 在本文中，我们通过提出记忆基础模型，向这一方向迈出了第一步，使基础模型具备原生记忆能力。 我们从两个角度形式化原生记忆：一是骨干网络内部持久且动态演化的记忆状态，二是通过模型计算自主存储和利用信息的原生记忆过程。 我们表明，原生记忆在架构、端到端优化和效率方面具有优势。

Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation.

基于这一形式化定义，我们提出 Metis，首个记忆基础模型原型。 Metis 引入一种新架构，为基础模型配备原生记忆状态，使历史信息能够被压缩进模型，并通过记忆注意力访问。 我们构建大规模记忆专用训练数据，并引入多种优化目标，通过中期训练获得这些原生记忆过程。 Metis 的在线记忆维护无需梯度，记忆更新只需要一次前向传播。 在推理时，所有已学习的模型权重保持冻结，而原生记忆状态通过标准前向计算自主变换。

Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.

通过广泛实验，我们表明 Metis 展现出原生记忆能力，并进一步详细分析其优势、局限和行为。 为推动未来对记忆基础模型的研究，我们公开项目和模型检查点。

1. Introduction

In recent years, large foundation models have achieved rapid development, demonstrating significant performance across many aspects, such as language modeling, code generation, and complex reasoning. This provides a solid foundation for constructing AI agents, which enables them to handle more complex tasks. Beyond the reasoning capabilities of foundation models, memory is another critical capability of AI agents, responsible for retaining past information and leveraging it to support future inference. In most previous works, memory is implemented by a module external to foundation models, rather than being natively integrated into their architectures. Representative approaches use Retrieval-Augmented Generation (RAG) to retrieve relevant textual information and incorporate it into the prompt to facilitate inference.

近年来，大型基础模型快速发展，在语言建模、代码生成和复杂推理等多个方面展现出显著性能。 这为构建 AI 智能体奠定了坚实基础，使其能够处理更复杂的任务。 除基础模型的推理能力外，记忆也是 AI 智能体的另一项关键能力，负责保留过去的信息并利用这些信息支持未来推理。 在大多数以往工作中，记忆由基础模型外部的模块实现，而不是原生集成到其架构中。 代表性方法使用检索增强生成（RAG）检索相关文本信息，并将其纳入提示以辅助推理。

However, external memory suffers from several limitations presented in Figure 1. First, external memory is decoupled from backbones with separated targets and processing stages. External memory typically aims to construct an informative context as input, and backbones only perform conditional language modeling over the constructed context. Therefore, external memory may not provide the most useful information to support the backbone inference, and the backbone may not utilize the memory optimally. Second, end-to-end optimization is difficult for external memory because gradients cannot be effectively propagated through discrete memory operations.

然而，外部记忆存在图1所示的若干局限。 第一，外部记忆与骨干网络相互解耦，目标和处理阶段彼此分离。 外部记忆通常旨在构建信息丰富的上下文作为输入，而骨干网络只在构建出的上下文上执行条件语言建模。 因此，外部记忆可能无法提供最有助于骨干网络推理的信息，骨干网络也可能无法以最优方式利用记忆。 第二，由于梯度无法有效穿过离散记忆操作传播，外部记忆难以进行端到端优化。

As a result, performing domain-specific post-training becomes highly challenging. Although some RL-based strategies can partially alleviate this issue by optimizing memory operations with reward signals, they suffer from efficiency issues. Finally, external memory requires additional explicit operations over the storage outside backbones, which inevitably increases the online inference latency.

因此，执行特定领域的后训练变得非常困难。 尽管一些基于强化学习的策略能够通过奖励信号优化记忆操作，从而部分缓解这一问题，但它们存在效率问题。 最后，外部记忆需要在骨干网络之外的存储上执行额外的显式操作，这不可避免地增加在线推理延迟。

To address the limitations of external memory, we introduce memory foundation models that empower large foundation models with native memory. It converts memory from an external module into an internal mechanism of backbones, directly involved in forward computation. Specifically, memory foundation models can generate responses based on the input instructions and their native memory, with autonomous memory transformation. We define the native memory from two critical aspects:

为解决外部记忆的局限，我们提出赋予大型基础模型原生记忆的记忆基础模型。 它把记忆从外部模块转化为骨干网络的内部机制，直接参与前向计算。 具体而言，记忆基础模型可以根据输入指令和自身的原生记忆生成响应，同时自主变换记忆。 我们从两个关键方面定义原生记忆：

• Native Memory State. Unlike traditional large foundation models, memory foundation models are natively stateful across multiple inferences, which can formulate, maintain, and utilize memory states inside backbones from prior inferences. Their memory states are dynamically represented as part of the parameters of backbones, whose semantic spaces are aligned at the pre-training or mid-training stage.
• Native Memory Procedure. Unlike memory engineering, memory foundation models natively integrate memory procedures within their inferences. Specific memory operations, such as remembering, forgetting, and updating, are accomplished autonomously alongside the backbone's forward computation, which impacts the native memory state based on input instructions.

• 原生记忆状态。 与传统大型基础模型不同，记忆基础模型在多次推理之间原生保持状态，能够根据先前推理在骨干网络内部形成、维护和利用记忆状态。其记忆状态以骨干网络参数的一部分动态表示，而这些参数的语义空间在预训练或中期训练阶段完成对齐。
• 原生记忆过程。 与记忆工程不同，记忆基础模型在推理内部原生集成记忆过程。记住、遗忘和更新等特定记忆操作会伴随骨干网络的前向计算自主完成，并依据输入指令影响原生记忆状态。

Memory foundation models aim to internalize memory capability into the model's forward computation. The memory state can be represented as dynamic parameters of backbones, and memory procedures are executed through computation. Therefore, like general foundation models, memory foundation models can be optimized and adapted to specific domains in a data-driven manner through post-training. This transformation is similar to the evolution from large foundation models to large reasoning models, where Chain-of-Thought (CoT) is natively integrated into inference to improve performance and efficiency. In addition, because native memory procedures can be integrated into the model's computation, they provide a foundation for improving the parallel efficiency of memory processing.

记忆基础模型旨在把记忆能力内化到模型的前向计算中。 记忆状态可以表示为骨干网络的动态参数，而记忆过程则通过计算执行。 因此，与通用基础模型一样，记忆基础模型可以通过后训练以数据驱动的方式针对特定领域进行优化和适配。 这一转变类似于从大型基础模型向大型推理模型的演进，其中思维链（CoT）被原生集成进推理，以提高性能和效率。 此外，由于原生记忆过程能够集成进模型计算，它们为提高记忆处理的并行效率奠定了基础。

图1：从外部记忆到原生记忆。

In this paper, we implement the first prototype of memory foundation models, named Metis. We design a new model architecture that has a native memory state inspired by Fast Weight Programming (FWP), which can be integrated into the backbone computation through memory attention. Specifically, we propose the Metis blocks as the basic units for native memory. Each of them primarily consists of a hyper memory block and a local memory block. In addition, we empower Metis with native memory procedures by designing specific optimization objectives, including memory reconstruction and memory operation objectives.

在本文中，我们实现了首个记忆基础模型原型，称为 Metis。 我们设计了一种受快速权重编程（FWP）启发、具有原生记忆状态的新模型架构，它能够通过记忆注意力集成到骨干网络计算中。 具体而言，我们提出 Metis 块作为原生记忆的基本单元。 每个 Metis 块主要由一个超记忆块和一个局部记忆块组成。 此外，我们通过设计包括记忆重构目标和记忆操作目标在内的特定优化目标，使 Metis 具备原生记忆过程。

These two objectives correspond to the compression upper limit of memory states and operation targets. We also design a regularization objective to improve robustness in complex scenarios. To support this training, we synthesize large-scale memory-specific data from publicly available datasets, enabling Metis to acquire native memory capabilities through mid-training. Finally, we conduct extensive experiments to demonstrate the effectiveness of our proposed framework, and explore more aspects for analysis.

这两个目标分别对应记忆状态的压缩上限和操作目标。 我们还设计了一个正则化目标，以提高复杂场景中的鲁棒性。 为支持这一训练，我们从公开数据集中合成大规模记忆专用数据，使 Metis 能够通过中期训练获得原生记忆能力。 最后，我们开展广泛实验以证明所提框架的有效性，并从更多方面进行分析。

From a general perspective, a fundamental problem of memory results from the time-streaming property of online information. At the storage stage, memory systems cannot determine how the received information will be used in the future. At the inference stage, the original information is no longer accessible, and only the stored information can be utilized. Therefore, memory can be considered as a prediction problem, where the model predicts how received information will be utilized in the future. Like other prediction tasks in machine learning, memory capability can also be acquired at the pre-training stage and generalized to other domains, and memory foundation models can provide the architectural and optimization foundation.

从一般视角看，记忆的一个根本问题源于在线信息的时间流特性。 在存储阶段，记忆系统无法确定接收到的信息未来将如何使用。 在推理阶段，原始信息已经无法访问，只能利用被存储的信息。 因此，记忆可以被视为一个预测问题，即模型预测接收到的信息未来将如何被利用。 与机器学习中的其他预测任务一样，记忆能力也可以在预训练阶段获得并泛化到其他领域，而记忆基础模型能够提供架构和优化基础。

Despite their promising properties, implementing memory foundation models remains highly challenging because their final goal is to completely eliminate the reliance on external memory in contexts. While Metis achieves great performance in memory-related tasks, it still faces several limitations. First, its performance degrades on long-term tasks, due to the information loss when compressed into fixed-size parameters. Second, it exhibits information confusion in some cases, possibly caused by the blending of semantics within the latent space. Despite these limitations, Metis provides a potential pathway to achieve memory foundation models. To benefit both the research community and industry, we release our project at https://github.com/MemTensor/Metis.

尽管记忆基础模型展现出可期的特性，但实现它仍极具挑战，因为其最终目标是完全消除对上下文中外部记忆的依赖。 尽管 Metis 在记忆相关任务上取得了出色性能，它仍面临若干局限。 第一，由于信息在压缩进固定大小的参数时会发生损失，其在长期任务上的性能会下降。 第二，它在某些情况下会出现信息混淆，这可能由潜在空间中的语义混合造成。 尽管存在这些局限，Metis 仍为实现记忆基础模型提供了一条潜在路径。 为使研究界和产业界共同受益，我们公开了项目：https://github.com/MemTensor/Metis。

Our contributions are summarized as follows:

我们的贡献总结如下：

• We introduce memory foundation models and native memory with formal definitions, providing further analysis from the perspective of native memory state and native memory procedure.
• We propose the first prototype of memory foundation models, named Metis, which is implemented with novel memory architectures and optimization tasks.
• We conduct extensive experiments to verify the effectiveness of our model, followed by detailed studies from multiple perspectives. We also publicly release our project to benefit the research community and industry.

• 我们为记忆基础模型和原生记忆给出形式化定义，并从原生记忆状态和原生记忆过程的角度进行进一步分析。
• 我们提出首个记忆基础模型原型 Metis，并以新颖的记忆架构和优化任务加以实现。
• 我们开展广泛实验以验证模型的有效性，随后从多个角度进行详细研究。我们还公开项目，以使研究界和产业界受益。

The rest of our paper is organized as follows. Section 2 provides the formal definition of memory foundation models. Section 3 details the model architecture of Metis. After that, we introduce our data construction pipeline in Section 4 and outline the optimization in Section 5. Extensive experimental results and analysis are presented in Section 6. Finally, we review related work in Section 7 and conclude in Section 8.

本文其余部分组织如下。 第 2 节给出记忆基础模型的形式化定义。 第 3 节详细介绍 Metis 的模型架构。 随后，我们在第 4 节介绍数据构建流程，并在第 5 节概述优化方法。 第 6 节给出广泛的实验结果与分析。 最后，我们在第 7 节回顾相关工作，并在第 8 节总结全文。

2. Memory Foundation Model

In this section, we provide a formal definition of the memory foundation model. Then, we introduce native memory from the perspectives of the native memory state and procedure. After that, we compare memory foundation models with previous works. Finally, we further discuss memory foundation models from the perspectives of lifelong learning and the evolving trends of AI agents.

在本节中，我们给出记忆基础模型的形式化定义。 随后，我们从原生记忆状态和原生记忆过程的角度介绍原生记忆。 然后，我们将记忆基础模型与以往工作进行比较。 最后，我们从终身学习和 AI 智能体演进趋势的角度进一步讨论记忆基础模型。

2.1 Definition

We define the memory foundation model under the multi-step scenario. Let a continuous interaction process be formulated as a sequence of discrete time steps $t\in \{1,2,\dots ,T\}$. At each step $t$, the foundation model receives an input instruction sequence denoted as $X_t$ and generates a response sequence $Y_t$.

我们在多步场景下定义记忆基础模型。 将连续交互过程形式化为离散时间步序列 $t\in \{1,2,\dots ,T\}$。 在每个时间步 $t$，基础模型接收记为 $X_t$ 的输入指令序列，并生成响应序列 $Y_t$。

For traditional foundation models without memory, the generation relies entirely on the current input context. The autoregressive decoding of the $k$-th token in the response is typically expressed as $y_{t,k}\sim P(y\mid X_t,Y_{t,<k};\theta )$, where $Y_{t,<k}$ denotes the previously generated tokens at step $t$, and $\theta$ represents the fixed parameters of the backbone. For foundation models with external memory, the autoregressive decoding process is then conditioned on the context $C_t$ alongside the input instruction. It can be expressed by $y_{t,k}\sim P(y\mid C_t,X_t,Y_{t,<k};\theta )$, where $C_t$ can be obtained from prior information $\{(X_i,Y_i){\}}_{i=1}^{t-1}$.

对于没有记忆的传统基础模型，生成完全依赖当前输入上下文。 响应中第 $k$ 个 token 的自回归解码通常表示为 $y_{t,k}\sim P(y\mid X_t,Y_{t,<k};\theta )$，其中 $Y_{t,<k}$ 表示时间步 $t$ 中此前生成的 token，$\theta$ 表示骨干网络的固定参数。 对于具有外部记忆的基础模型，自回归解码过程还以输入指令和上下文 $C_t$ 为条件。 它可以表示为 $y_{t,k}\sim P(y\mid C_t,X_t,Y_{t,<k};\theta )$，其中 $C_t$ 可以从先前信息 $\{(X_i,Y_i){\}}_{i=1}^{t-1}$ 中获得。

The external memory framework commonly has two explicit procedures, including the storage procedure ${\mathbf{C}}_t={\mathbf{C}}_{t-1}\oplus \{(X_t,Y_t)\}$, and the retrieval procedure $C_t={\mathbf{C}}_{t-1}\otimes X_t$. Here, $\oplus$ denotes the general writing operation, and $\otimes$ represents the general reading operation with the textual memory storage ${\mathbf{C}}_{t-1}$. Both of them are executed outside the model inference process.

外部记忆框架通常包含两个显式过程，即存储过程 ${\mathbf{C}}_t={\mathbf{C}}_{t-1}\oplus \{(X_t,Y_t)\}$ 和检索过程 $C_t={\mathbf{C}}_{t-1}\otimes X_t$。 这里，$\oplus$ 表示通用写入操作，$\otimes$ 表示基于文本记忆存储 ${\mathbf{C}}_{t-1}$ 的通用读取操作。 二者都在模型推理过程之外执行。

Definition 1 (Memory Foundation Model). The memory foundation model is defined as an autoregressive foundation model empowered by native memory across multi-step interactions. At each step $t$, the generation of the $k$-th token is conditioned on the input instruction $X_t$ and the previously generated tokens $Y_{t,<k}$ by

定义 1（记忆基础模型）。 记忆基础模型被定义为在多步交互中由原生记忆赋能的自回归基础模型。 在每个时间步 $t$，第 $k$ 个 token 的生成以输入指令 $X_t$ 和此前生成的 token $Y_{t,<k}$ 为条件：

$$y_{t,k}\sim P(y\mid X_t,Y_{t,<k};{\theta }_t).$$

Here the model parameter ${\theta }_t$ integrates information from previous steps into its native parametric space (i.e., native memory state). Concurrently, ${\theta }_{t+1}$ is autonomously transformed during the forward computation inside the model based on the input instruction $X_t$ and output $Y_t$ (i.e., native memory procedure).

这里，模型参数 ${\theta }_t$ 把先前时间步的信息整合进其原生参数空间（即原生记忆状态）。 与此同时，${\theta }_{t+1}$ 在模型内部的前向计算期间依据输入指令 $X_t$ 和输出 $Y_t$ 自主变换（即原生记忆过程）。

In contrast, the memory foundation model internalizes memory into the backbone's computation, which is empowered with native memory. Instead of relying on an external explicit storage ${\mathbf{C}}_t$ and context $C_t$, it maintains a native memory state, which acts as dynamic parameters across multiple steps. In addition, rather than explicitly executing memory procedures outside backbones, memory procedures in the memory foundation model occur autonomously alongside the model's forward computation, such as operations like remembering, forgetting, and updating.

相比之下，记忆基础模型把记忆内化到骨干网络计算中，从而获得原生记忆。 它不依赖外部显式存储 ${\mathbf{C}}_t$ 和上下文 $C_t$，而是维护一个在多个时间步之间充当动态参数的原生记忆状态。 此外，记忆基础模型不在骨干网络之外显式执行记忆过程；记住、遗忘和更新等操作会伴随模型前向计算自主发生。

2.2 Native Memory State

In this paper, we adopt a strict definition for the source of memory. We only consider the information acquired during online interactions as memory, where information available before the interaction starts is excluded. In fact, such offline information is better viewed as knowledge rather than memory, because it does not contain trajectory-specific information for personalization and does not require real-time adaptation.

在本文中，我们对记忆来源采用严格定义。 我们只把在线交互期间获得的信息视为记忆，而排除交互开始前已经可用的信息。 事实上，这类离线信息更适合被视为知识而非记忆，因为它不包含用于个性化的轨迹特定信息，也不需要实时适配。

Since the stored information varies across different steps, the parameters ${\theta }_t$ cannot remain completely static. Consequently, at least a portion of the parameters must change dynamically according to the input, and we denote this dynamic part as the memory state ${\mathbf{M}}_t$. In the memory foundation model, the representation of stored information is supposed to be coupled with the backbone to participate in forward computation. Therefore, the native memory state should be represented in parametric form.

由于不同时间步存储的信息会发生变化，参数 ${\theta }_t$ 不能保持完全静态。 因此，至少一部分参数必须依据输入动态变化，我们把这一动态部分记为记忆状态 ${\mathbf{M}}_t$。 在记忆基础模型中，存储信息的表示应与骨干网络耦合并参与前向计算。 因此，原生记忆状态应采用参数形式表示。

Although textual memory offers advantages in interpretability and cross-model compatibility, its discrete representation results in low information density and requires repetitive prefilling. In contrast, parametric memory represents prior information in a dense form, which increases the efficiency of storage and utilization.

尽管文本记忆在可解释性和跨模型兼容性方面具有优势，其离散表示会导致信息密度低，并需要重复预填充。 相比之下，参数记忆以稠密形式表示先前信息，从而提高存储和利用效率。

In addition, the semantic spaces of both dynamic parameters ${\mathbf{M}}_t$ and static parameters $\mathrm{\Phi }={\theta }_t\setminus {\mathbf{M}}_t$ must be aligned during the pre-training or mid-training stage before conducting online inference. This alignment enables the dynamic parameters at different steps to compute collaboratively with the static parameters. During online interactions, the native memory state can be updated and utilized through the native memory procedure, which empowers the memory foundation model with statefulness across different steps.

此外，在进行在线推理前，动态参数 ${\mathbf{M}}_t$ 和静态参数 $\mathrm{\Phi }={\theta }_t\setminus {\mathbf{M}}_t$ 的语义空间都必须在预训练或中期训练阶段完成对齐。 这种对齐使不同时间步的动态参数能够与静态参数协同计算。 在在线交互期间，原生记忆状态可以通过原生记忆过程进行更新和利用，从而使记忆基础模型在不同时间步之间保持状态。

2.3 Native Memory Procedure

In terms of memory, storage and utilization are two core procedures to handle online information with the time-streaming property. Memory storage retains past information, while memory utilization leverages this stored information to support model inference. They aim to address the temporal mismatch between information supply and usage.

对于记忆而言，存储和利用是处理具有时间流特性的在线信息的两个核心过程。 记忆存储保留过去的信息，而记忆利用则借助这些存储的信息支持模型推理。 二者旨在解决信息供给与使用之间的时间错配。

The memory storage procedure typically involves several specific operations, such as remembering, forgetting, and updating. From the perspective of foundation models, an input instruction contains both the intent and the content of information processing. For instance, “Alice is 24 years old” implies remembering her age, while “Bob moved from London to Boston” indicates updating his location. A native storage procedure should directly map the input instruction to the update value of the memory state.

记忆存储过程通常涉及记住、遗忘和更新等若干具体操作。 从基础模型的角度看，一条输入指令同时包含信息处理的意图和内容。 例如，*“Alice 今年 24 岁”意味着记住她的年龄，而“Bob 从伦敦搬到波士顿”*表示更新他的位置。 原生存储过程应把输入指令直接映射为记忆状态的更新值。

In contrast, external memory relies on rule-based and predefined operations to handle its intent and content separately. Although most operations can be categorized into insertion, deletion, and modification, the semantic intent and content cannot be easily decoupled into discrete rules. In fact, the storage procedure essentially predicts how current information will be used in the future. Because rule-based procedures operate in a discrete function space, they struggle to achieve optimal prediction performance.

相比之下，外部记忆依赖基于规则的预定义操作，分别处理意图和内容。 尽管大多数操作可以归类为插入、删除和修改，但语义意图与内容很难被解耦为离散规则。 事实上，存储过程本质上是在预测当前信息未来将如何使用。 由于基于规则的过程在离散函数空间中运行，它们难以获得最优预测性能。

The primary goal of the memory utilization procedure is to assist inference with the stored information. From the foundation model perspective, it can be considered as letting the required information of the input instruction participate in the forward computation. For example, answering “Where does Bob live now?” requires previously stored living information to facilitate inference. Thus, a native memory utilization procedure should directly map the input instruction and memory state to the generated output.

记忆利用过程的首要目标是借助已存储信息辅助推理。 从基础模型的角度看，可以把它理解为让输入指令所需的信息参与前向计算。 例如，回答*“Bob 现在住在哪里？”*需要此前存储的居住信息来辅助推理。 因此，原生记忆利用过程应把输入指令和记忆状态直接映射为生成输出。

External memory designs rules to trigger retrieval, reranking, and concatenation. However, the information requirement cannot be defined and captured by discrete and finite rules. For example, an instruction may require information based on semantic similarity, emotion, or even complex combinations of implicit metrics. The memory utilization procedure predicts the information requirements, which is coupled with the inference process. Therefore, it should not be divided into discrete stages limited by discrete function spaces.

外部记忆设计规则来触发检索、重排序和拼接。 然而，离散且有限的规则无法定义和捕捉信息需求。 例如，一条指令可能需要基于语义相似性、情感，甚至隐式指标的复杂组合来获取信息。 记忆利用过程预测信息需求，并与推理过程相耦合。 因此，它不应被划分为受离散函数空间限制的多个离散阶段。

Consequently, the memory procedure should be modeled within a continuous function space and implemented via numerical computation, which is tightly coupled with the forward computation of the backbone. In a memory foundation model, the native memory procedure autonomously executes both memory storage and utilization during the forward computation. This native memory procedure should be established during the pre-training or mid-training stage. In addition, this memory procedure paradigm has significant advantages in both efficiency and end-to-end optimization.

因此，记忆过程应在连续函数空间中建模，并通过与骨干网络前向计算紧密耦合的数值计算实现。 在记忆基础模型中，原生记忆过程会在前向计算期间自主执行记忆存储和利用。 这种原生记忆过程应在预训练或中期训练阶段建立。 此外，这一记忆过程范式在效率和端到端优化方面都具有显著优势。

2.4 Comparison with Previous Works

Test-time Training. Memory foundation models differ from test-time training (TTT) in three key aspects. First, TTT typically adapts the model within a single sequence, where the dynamic parameters are updated to better fit the current input. In contrast, memory foundation models are defined under multi-step interactions. Their dynamic parameters serve as persistent native memory states that store information from previous steps and support future inference.

测试时训练。 记忆基础模型与测试时训练（TTT）在三个关键方面存在差异。 第一，TTT 通常在单个序列内适配模型，更新动态参数以更好地拟合当前输入。 相比之下，记忆基础模型是在多步交互下定义的。 其动态参数充当持久的原生记忆状态，存储先前时间步的信息并支持未来推理。

Second, TTT does not explicitly provide native memory procedures. Its update is usually driven by self-supervised language modeling, which helps the model absorb prior information within the current sequence. However, it does not specify how the model should semantically remember, forget, update, or reflect on information according to input instructions. In contrast, memory foundation models are trained with memory reconstruction and operation objectives, enabling the model to autonomously execute semantic memory operations in the latent parametric space and transform the native memory state accordingly.

第二，TTT 并不显式提供原生记忆过程。 其更新通常由自监督语言建模驱动，帮助模型吸收当前序列中的先前信息。 然而，它没有规定模型应如何依据输入指令在语义层面记住、遗忘、更新或反思信息。 相比之下，记忆基础模型通过记忆重构和操作目标进行训练，使模型能够在潜在参数空间中自主执行语义记忆操作，并相应地变换原生记忆状态。

Third, many TTT methods are motivated by efficient long-context modeling, and they often introduce recurrent layers to replace full attention. However, memory foundation models pursue a different goal. They do not aim to replace the standard full-attention computation within the current step. Instead, they introduce information from previous interaction steps through native memory states as residuals. In summary, TTT is primarily a mechanism for inference-time adaptation, while memory foundation models formulate memory as a persistent, instruction-driven, and procedure-aware capability of foundation models.

第三，许多 TTT 方法以高效长上下文建模为动机，通常引入循环层来替代完整注意力。 然而，记忆基础模型追求不同的目标。 它们并不旨在替代当前时间步内的标准完整注意力计算。 相反，它们通过原生记忆状态以残差形式引入先前交互时间步的信息。 总之，TTT 主要是一种推理时适配机制，而记忆基础模型把记忆形式化为基础模型持久、指令驱动且过程感知的能力。

Memory-Augmented Neural Networks. Memory-augmented neural networks (MANNs) introduce additional memory modules to neural models, such as differentiable memory slots and learned read-write operations. These models show that neural networks can store external information and retrieve it for later computation. Nevertheless, memory foundation models differ in how memory is integrated with the backbone. In MANNs, the memory module is usually a separate storage component controlled by a neural controller. Although the operations can be differentiable, the memory is still external to the main model parameters, which are often designed independently from the backbone.

记忆增强神经网络。 记忆增强神经网络（MANN）为神经模型引入额外记忆模块，例如可微记忆槽和学习得到的读写操作。 这些模型表明，神经网络可以存储外部信息并在后续计算中检索它。 尽管如此，记忆基础模型在记忆与骨干网络的集成方式上有所不同。 在 MANN 中，记忆模块通常是由神经控制器控制的独立存储组件。 尽管操作可以是可微的，记忆仍位于主要模型参数之外，而且往往独立于骨干网络进行设计。

In contrast, memory foundation models internalize memory into the backbone computation. The memory state is represented in a parametric form and participates directly in forward computation. The memory procedure is also modeled by the same continuous function space as the backbone, rather than being implemented as a separate controller over explicit slots. Therefore, memory foundation models can be regarded as a step from externally augmented memory toward native memory inside foundation models.

相比之下，记忆基础模型把记忆内化到骨干网络计算中。 记忆状态以参数形式表示，并直接参与前向计算。 记忆过程也由与骨干网络相同的连续函数空间建模，而不是实现为控制显式槽位的独立控制器。 因此，记忆基础模型可以被视为从外部增强记忆迈向基础模型内部原生记忆的一步。

Other Methods. Compared with In-place TTT, MemGen and $\delta$-Mem, which still rely on textual memories in the context and use additional latent summaries to improve inference, memory foundation models remove textual memory from the context entirely. Compared with MEMO and MemFT, which primarily handle offline documents, memory foundation models focus on test-time information. Compared with Memory${}^3$, which takes an important step beyond textual RAG by encoding knowledge into retrievable explicit memories, memory foundation models further extend this direction toward native memory.

其他方法。 In-place TTT、MemGen 和 $\delta$-Mem 仍依赖上下文中的文本记忆，并使用额外的潜在摘要改进推理；相比之下，记忆基础模型完全移除了上下文中的文本记忆。 MEMO 和 MemFT 主要处理离线文档，而记忆基础模型聚焦于测试时信息。 Memory${}^3$ 通过把知识编码为可检索的显式记忆，迈出了超越文本 RAG 的重要一步；记忆基础模型则进一步沿这一方向迈向原生记忆。

While Memory${}^3$ primarily constructs explicit memories from offline corpora and retrieves them to augment attention computation, memory foundation models internalize information acquired from online interactions as persistent dynamic states within the backbone. They further enable these states to be autonomously maintained and transformed through native memory procedures across multiple interaction steps.

Memory${}^3$ 主要从离线语料构建显式记忆，并检索这些记忆以增强注意力计算；记忆基础模型则把在线交互中获得的信息内化为骨干网络中的持久动态状态。 它们还使这些状态能够通过原生记忆过程在多个交互时间步之间被自主维护和变换。

2.5 Discussion

For memory foundation models, the onset of interaction represents a key transition from static to dynamic knowledge acquisition. Knowledge acquired before interaction originates from offline pre-training and is retained in static parameters, while information received during interaction is acquired at test time and stored in dynamic memory states. Therefore, ${\theta }_1$ can also serve as initial supplementary information outside of pre-training. Because ${\theta }_1$ captures transferable domain knowledge, models deployed in similar domains can be initialized with the same ${\theta }_1$ to provide baseline information.

对于记忆基础模型，交互的开始代表从静态知识获取向动态知识获取的关键转变。 交互前获得的知识源自离线预训练并保留在静态参数中，而交互期间接收的信息在测试时获得并存入动态记忆状态。 因此，${\theta }_1$ 也可以充当预训练之外的初始补充信息。 由于 ${\theta }_1$ 捕捉了可迁移的领域知识，部署在相似领域中的模型可以使用相同的 ${\theta }_1$ 初始化，以提供基础信息。

Moreover, native memory aligns with the evolving trend of foundation models. Inspired by large reasoning models, we find that an agent's external capabilities can be expressed natively by the foundation model through optimization. In other words, supervised data of target behaviors can activate internal capabilities and generalize them to other tasks. These native capabilities can provide advantages in generalization, efficiency, and optimization properties. Memory is also a critical agent capability that traditionally relies on external modules.

此外，原生记忆符合基础模型的演进趋势。 受大型推理模型启发，我们发现智能体的外部能力可以通过优化由基础模型原生表达。 换言之，目标行为的监督数据可以激活内部能力，并使其泛化到其他任务。 这些原生能力可以在泛化、效率和优化性质方面带来优势。 记忆同样是智能体的一项关键能力，传统上依赖外部模块。

Therefore, we argue that memory can also be natively triggered through memory-specific tasks. However, unlike reasoning, which is purely a process, memory also involves a storage entity. It requires us to modify the model architecture to incorporate a storage entity as the memory state. Then, both the memory state and procedures are supposed to be modeled under a collaborative function space. This enables us to empower foundation models with memory capabilities via an optimization-driven approach.

因此，我们认为记忆也可以通过记忆专用任务被原生触发。 然而，与纯粹作为过程的推理不同，记忆还涉及存储实体。 这要求我们修改模型架构，把存储实体作为记忆状态纳入其中。 随后，记忆状态和记忆过程都应在协同函数空间中建模。 这使我们能够通过优化驱动的方法赋予基础模型记忆能力。

3. Metis Architecture

In this section, we first present some preliminaries. Then, we introduce Metis as the prototype of memory foundation models with the Metis block. After that, we demonstrate how this architecture supports native memory storage and utilization procedures through computation. Finally, we provide theoretical insights, theoretical error analysis, and further discussions. The overview of the Metis framework is presented in Figure 2.

在本节中，我们首先介绍一些预备知识。 随后，我们介绍作为记忆基础模型原型的 Metis 及其 Metis 块。 之后，我们说明这种架构如何通过计算支持原生记忆存储与利用过程。 最后，我们给出理论见解、理论误差分析和进一步讨论。 Metis 框架概览如图2所示。

3.1 Preliminaries

We adopt causal language models as the primary implementation of memory foundation models, as they are dominantly used in modern foundation models. We present the standard architecture of causal language models, which primarily consists of $N$ stacked Transformer blocks followed by a language modeling head.

我们采用因果语言模型作为记忆基础模型的主要实现，因为它们在现代基础模型中占据主导地位。 我们给出因果语言模型的标准架构，它主要由 $N$ 个堆叠的 Transformer 块和其后的语言建模头组成。

图2：Metis 架构概览。

Transformer Block. To highlight the core architecture, we focus on causal self-attention and the feed-forward network (FFN), which are major components of modern Transformers. We omit other details, such as positional embeddings, multi-head attention strategies, and hybrid attention mechanisms, as they can be directly incorporated into our framework.

Transformer 块。 为突出核心架构，我们关注现代 Transformer 的主要组成部分：因果自注意力和前馈网络（FFN）。 我们省略位置嵌入、多头注意力策略和混合注意力机制等其他细节，因为它们可以直接纳入我们的框架。

We denote the input of the $l$-th Transformer block as ${\mathbf{H}}^{(l-1)}\in {\mathbb{R}}^{L\times d}$, where $L$ is the sequence length and $d$ is the dimension of hidden states. After applying the pre-normalization function $\mathrm{PreNorm}$, we obtain ${\tilde{\mathbf{H}}}^{(l)}=\mathrm{PreNorm}({\mathbf{H}}^{(l-1)})$ as the input of causal self-attention. We denote ${\mathbf{W}}_Q^{(l)},{\mathbf{W}}_K^{(l)}\in {\mathbb{R}}^{d\times d_k},{\mathbf{W}}_V^{(l)}\in {\mathbb{R}}^{d\times d_v}$ as query, key, and value projection matrices of this layer. Then, we obtain the query state, key state, and value state of the $l$-th layer by

我们将第 $l$ 个 Transformer 块的输入表示为 ${\mathbf{H}}^{(l-1)}\in {\mathbb{R}}^{L\times d}$，其中 $L$ 是序列长度，$d$ 是隐状态维度。 应用预归一化函数 $\mathrm{PreNorm}$ 后，我们得到 ${\tilde{\mathbf{H}}}^{(l)}=\mathrm{PreNorm}({\mathbf{H}}^{(l-1)})$，作为因果自注意力的输入。 我们将 ${\mathbf{W}}_Q^{(l)},{\mathbf{W}}_K^{(l)}\in {\mathbb{R}}^{d\times d_k},{\mathbf{W}}_V^{(l)}\in {\mathbb{R}}^{d\times d_v}$ 表示为该层的查询、键和值投影矩阵。 随后，我们通过下式得到第 $l$ 层的查询状态、键状态和值状态：

$$\begin{array}{rl}{\mathbf{Q}}^{(l)}& ={\tilde{\mathbf{H}}}^{(l)}{\mathbf{W}}_Q^{(l)},\\ {\mathbf{K}}^{(l)}& ={\tilde{\mathbf{H}}}^{(l)}{\mathbf{W}}_K^{(l)},\\ {\mathbf{V}}^{(l)}& ={\tilde{\mathbf{H}}}^{(l)}{\mathbf{W}}_V^{(l)}.\end{array}$$

After that, the causal self-attention can be calculated by

之后，因果自注意力可计算为：

$$\begin{array}{}\text{(1)}& {\mathbf{A}}^{(l)}=\mathrm{Softmax}(\frac{{\mathbf{Q}}^{(l)}({\mathbf{K}}^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L)){\mathbf{V}}^{(l)}.\end{array}$$

where $d_k$ is the attention head dimension, and $\mathrm{Mask}(L)$ is the causal mask defined as $\mathrm{Mask}(L{)}_{i,j}=-\mathrm{\infty }$ if $j>i$, and $0$ otherwise. Then, it adds the projected output attention to the residual after projection by

其中，$d_k$ 是注意力头维度，$\mathrm{Mask}(L)$ 是因果掩码：当 $j>i$ 时，$\mathrm{Mask}(L{)}_{i,j}=-\mathrm{\infty }$，否则为 $0$。 随后，它通过下式将投影后的注意力输出加到残差上：

$${\mathbf{H}}^{\prime (l)}={\mathbf{H}}^{(l-1)}+{\mathbf{A}}^{(l)}{\mathbf{W}}_O^{(l)}.$$

where ${\mathbf{W}}_O^{(l)}\in {\mathbb{R}}^{d_v\times d}$. Finally, ${\mathbf{H}}^{\prime (l)}$ is passed through FFN with a residual connection to get the $l$-th layer output by

其中 ${\mathbf{W}}_O^{(l)}\in {\mathbb{R}}^{d_v\times d}$。 最后，${\mathbf{H}}^{\prime (l)}$ 通过带残差连接的 FFN，以下式得到第 $l$ 层输出：

$${\mathbf{H}}^{(l)}={\mathbf{H}}^{\prime (l)}+\mathrm{FFN}(\mathrm{Norm}({\mathbf{H}}^{\prime (l)})).$$

where the activation ${\mathbf{H}}^{(l)}$ is also the input of $(l+1)$-th layer.

其中，激活 ${\mathbf{H}}^{(l)}$ 同时也是第 $(l+1)$ 层的输入。

Causal Language Model. We denote the sequence of input tokens as $X=(x_1,x_2,\dots ,x_L)$. A causal language model first maps these discrete tokens into continuous vector representations. Let $\mathbf{E}\in {\mathbb{R}}^{|\mathcal{V}|\times d}$ denote the token embedding matrix, where $|\mathcal{V}|$ is the vocabulary size. The initial hidden state ${\mathbf{H}}^{(0)}\in {\mathbb{R}}^{L\times d}$ is obtained by extracting the corresponding embeddings and combining them with positional embeddings.

因果语言模型。 我们将输入 token 序列表示为 $X=(x_1,x_2,\dots ,x_L)$。 因果语言模型首先将这些离散 token 映射为连续向量表示。 令 $\mathbf{E}\in {\mathbb{R}}^{|\mathcal{V}|\times d}$ 表示 token 嵌入矩阵，其中 $|\mathcal{V}|$ 是词表大小。 初始隐状态 ${\mathbf{H}}^{(0)}\in {\mathbb{R}}^{L\times d}$ 通过提取相应嵌入并将其与位置嵌入结合而得。

After that, this initial representation is processed sequentially through the stack of $N$ Transformer blocks by

之后，该初始表示通过 $N$ 个 Transformer 块的堆叠依次处理：

$${\mathbf{H}}^{(l)}={\mathrm{TransformerBlock}}^{(l)}({\mathbf{H}}^{(l-1)}),\text{for }l=1,2,\dots ,N.$$

Then, the final hidden state ${\mathbf{H}}^{(N)}$ represents the contextualized input, and the language modeling head maps this final state back to the vocabulary space to predict the probability distribution for the next token. This process is commonly modeled by a linear projection after normalization, followed by a softmax function

随后，最终隐状态 ${\mathbf{H}}^{(N)}$ 表示经上下文化的输入，语言建模头将这一最终状态映射回词表空间，以预测下一个 token 的概率分布。 该过程通常建模为归一化后的线性投影，紧接一个 softmax 函数：

$$P(\cdot \mid x_{\le i})=\mathrm{Softmax}(\mathrm{Norm}({\mathbf{H}}_{(i)}^{(N)}){\mathbf{W}}_{\mathrm{LM}}).$$

where ${\mathbf{H}}_{(i)}^{(N)}\in {\mathbb{R}}^d$ is the final hidden vector at position $i$, and ${\mathbf{W}}_{\mathrm{LM}}\in {\mathbb{R}}^{d\times |\mathcal{V}|}$ represents the projection matrix. After sampling $x_{i+1}\sim P(x_{i+1}\mid x_{\le i})$, this new token is appended to the sequence, and the model repeats the process until it decodes an end-of-sequence token or reaches the maximum length.

其中，${\mathbf{H}}_{(i)}^{(N)}\in {\mathbb{R}}^d$ 是位置 $i$ 处的最终隐向量，${\mathbf{W}}_{\mathrm{LM}}\in {\mathbb{R}}^{d\times |\mathcal{V}|}$ 表示投影矩阵。 对 $x_{i+1}\sim P(x_{i+1}\mid x_{\le i})$ 采样后，新 token 被追加到序列中，模型重复该过程，直到解码出序列结束 token 或达到最大长度。

During the pre-training phase, the causal language model is optimized using the standard autoregressive next-token prediction objective. It minimizes the negative log-likelihood of the training sequences by

在预训练阶段，因果语言模型使用标准的自回归下一 token 预测目标进行优化。 它通过下式最小化训练序列的负对数似然：

$$\theta =\arg \underset{\theta }{min}\sum _{X\in \mathcal{D}}\sum _{i=1}^{|X|-1}-\log P(x_{i+1}\mid x_{\le i};\theta ).$$

where $\mathcal{D}$ represents the pre-training corpus and $\theta$ encompasses all the trainable parameters of the model.

其中，$\mathcal{D}$ 表示预训练语料库，$\theta$ 包含模型的所有可训练参数。

3.2 Native Memory State

To implement the native memory state, we propose the Metis blocks inside Transformer blocks in Figure 2(b), where each Metis block consists of a local memory block and a hyper memory block in Figure 2(c). The local memory blocks are responsible for maintaining the dense representation of prior information, while the hyper memory blocks construct parametric function spaces for native memory procedures to transform memory states.

为实现原生记忆状态，我们在 Transformer 块内提出图2(b)所示的 Metis 块，每个 Metis 块由图2(c)所示的局部记忆块和超记忆块组成。 局部记忆块负责维护先前信息的稠密表示，超记忆块则为原生记忆过程构建参数化函数空间，以转换记忆状态。

Local Memory Block. Local memory blocks maintain the memory state at the current step, so we define a dense memory network ${\mathbf{M}}^{(l)}\in {\mathbb{R}}^{d_k\times d_v}$ inside the $l$-th local memory block. At the step $t$, we denote it as ${\mathbf{M}}_t^{(l)}$. The model also maintains a query-key normalization vector as ${\mathbf{S}}_t^{(l)}\in {\mathbb{R}}^{d_k}$. Both ${\mathbf{M}}^{(l)}$ and ${\mathbf{S}}_t^{(l)}$ are dynamic parameters updated across different steps. Specifically, we set ${\mathbf{M}}_1^{(l)}=\mathbf{0}$ and ${\mathbf{S}}_1^{(l)}=\mathbf{0}$ by default.

局部记忆块。 局部记忆块维护当前步骤的记忆状态，因此我们在第 $l$ 个局部记忆块中定义稠密记忆网络 ${\mathbf{M}}^{(l)}\in {\mathbb{R}}^{d_k\times d_v}$。 在步骤 $t$ 上，我们将其表示为 ${\mathbf{M}}_t^{(l)}$。 模型还维护查询—键归一化向量 ${\mathbf{S}}_t^{(l)}\in {\mathbb{R}}^{d_k}$。 ${\mathbf{M}}^{(l)}$ 和 ${\mathbf{S}}_t^{(l)}$ 都是在不同步骤间更新的动态参数。 具体而言，我们默认设置 ${\mathbf{M}}_1^{(l)}=\mathbf{0}$ 和 ${\mathbf{S}}_1^{(l)}=\mathbf{0}$。

Hyper Memory Block. Hyper memory blocks are responsible for updating the dynamic parameters in local memory blocks based on the intermediate activations of the current input $X_t$ and output $Y_t$. Each of them consists of static parameters obtained through mid-training, which remain unchanged during interactions. It serves as the parametric foundation of the native memory storage procedure. Specifically, each hyper memory block is parameterized by several optimizable parameters. First, it has a learnable importance vector ${\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}\in {\mathbb{R}}^d$, which scores the intermediate activations for adaptive aggregation.

超记忆块。 超记忆块负责根据当前输入 $X_t$ 和输出 $Y_t$ 的中间激活，更新局部记忆块中的动态参数。 每个超记忆块都由通过中期训练获得的静态参数组成，这些参数在交互期间保持不变。 它充当原生记忆存储过程的参数基础。 具体而言，每个超记忆块由若干可优化参数来参数化。 首先，它具有可学习的重要性向量 ${\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}\in {\mathbb{R}}^d$，用于对中间激活评分以进行自适应聚合。

In addition, we set the memory key and value projection matrices ${\tilde{\mathbf{W}}}_K^{(l)}\in {\mathbb{R}}^{d\times d_k}$ and ${\tilde{\mathbf{W}}}_V^{(l)}\in {\mathbb{R}}^{d\times d_v}$, which map the selected hidden states into the memory keys and values for the local memory. We also set the memory query projection matrix ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{d\times d_k}$ to reduce the error in the native memory utilization procedure.

此外，我们设置记忆键和值投影矩阵 ${\tilde{\mathbf{W}}}_K^{(l)}\in {\mathbb{R}}^{d\times d_k}$ 和 ${\tilde{\mathbf{W}}}_V^{(l)}\in {\mathbb{R}}^{d\times d_v}$，将选中的隐状态映射为局部记忆的记忆键和值。 我们还设置记忆查询投影矩阵 ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{d\times d_k}$，以减少原生记忆利用过程中的误差。

3.3 Native Memory Procedure

The native memory procedure consists of memory storage and utilization procedures, as we discuss in Section 2.3. In the native memory storage procedure of our framework, hyper memory blocks update local memory blocks as part of the model computation, based on intermediate activations. In the native memory utilization procedure, local memory blocks incorporate the current memory states into the forward computation.

如第 2.3 节所述，原生记忆过程由记忆存储和利用过程组成。 在我们框架的原生记忆存储过程中，超记忆块根据中间激活，作为模型计算的一部分更新局部记忆块。 在原生记忆利用过程中，局部记忆块将当前记忆状态纳入前向计算。

Native Memory Storage Procedure. After completing step $t$, we denote the input hidden states at the $l$-th layer as ${\mathbf{H}}_t^{(l-1)}$. Then, the hyper memory block aggregates it into a compact representation through a learned adaptive aggregation. Specifically, we first pre-normalize the hidden states as ${\tilde{\mathbf{H}}}_t^{(l)}=\mathrm{PreNorm}({\mathbf{H}}_t^{(l-1)})$ and score each of the $L$ tokens with a learnable importance vector ${\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}\in {\mathbb{R}}^d$, obtaining an importance distribution

原生记忆存储过程。 完成步骤 $t$ 后，我们将第 $l$ 层的输入隐状态表示为 ${\mathbf{H}}_t^{(l-1)}$。 随后，超记忆块通过学习到的自适应聚合，将其聚合为紧凑表示。 具体而言，我们首先将隐状态预归一化为 ${\tilde{\mathbf{H}}}_t^{(l)}=\mathrm{PreNorm}({\mathbf{H}}_t^{(l-1)})$，并使用可学习的重要性向量 ${\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}\in {\mathbb{R}}^d$ 对 $L$ 个 token 逐一评分，得到重要性分布：

$${\mathbf{p}}_t^{(l)}=\mathrm{Softmax}\left(\frac{{\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}}{\tau }\right)\in {\mathbb{R}}^L.$$

where $\tau$ is a temperature coefficient. Then, we obtain a subset of positions based on the top-$\rho$. We sort these probabilities in descending order as $p_{(1)}\ge p_{(2)}\ge \cdots \ge p_{(L)}$, and keep the smallest prefix whose cumulative value reaches the threshold $\rho$. The number of selected positions can be expressed by

其中 $\tau$ 是温度系数。 随后，我们基于 top-$\rho$ 得到一个位置子集。 我们将这些概率按降序排列为 $p_{(1)}\ge p_{(2)}\ge \cdots \ge p_{(L)}$，并保留累积值达到阈值 $\rho$ 的最短前缀。 选中位置数量可表示为：

$$L_t^{\prime }=\mathrm{clip}(min\{k:\sum _{r=1}^k{p}_{(r)}\ge \rho \},K_{min},L).$$

where $K_{min}$ denotes the minimum number of selected positions. These $L_t^{\prime }$ positions with the highest scores form the selected set ${\mathcal{S}}_t^{(l)}$, and we gather corresponding hidden states by

其中 $K_{min}$ 表示选中位置的最小数量。 得分最高的 $L_t^{\prime }$ 个位置构成选中集合 ${\mathcal{S}}_t^{(l)}$，我们通过下式汇集对应的隐状态：

$${\overline{\mathbf{H}}}_t^{(l)}={\mathbf{\Pi }}_t^{(l)}{\tilde{\mathbf{H}}}_t^{(l)}\in {\mathbb{R}}^{L_t^{\prime }\times d},\text{with }{L}_t^{\prime }\ll L.$$

where ${\mathbf{\Pi }}_t^{(l)}\in \{0,1{\}}^{L_t^{\prime }\times L}$, whose rows are the one-hot indicators of ${\mathcal{S}}_t^{(l)}$. Since the top-$\rho$ selection is non-differentiable, we adopt a straight-through estimator that routes the gradients through the dense distribution ${\mathbf{p}}_t^{(l)}$, making the scorer ${\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}$ end-to-end trainable. After that, we compute the projected memory key states and memory value states as

其中 ${\mathbf{\Pi }}_t^{(l)}\in \{0,1{\}}^{L_t^{\prime }\times L}$，它的各行是 ${\mathcal{S}}_t^{(l)}$ 的独热指示向量。 由于 top-$\rho$ 选择不可微，我们采用直通估计器，通过稠密分布 ${\mathbf{p}}_t^{(l)}$ 传递梯度，使评分器 ${\tilde{\mathbf{w}}}_{\text{agg}}^{(l)}$ 能够端到端训练。 之后，我们计算投影后的记忆键状态和记忆值状态：

$$\begin{array}{rl}{\tilde{\mathbf{K}}}_t^{(l)}& ={\overline{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_K^{(l)},\\ {\tilde{\mathbf{V}}}_t^{(l)}& ={\overline{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_V^{(l)}.\end{array}$$

Finally, the dense memory network is updated based on ${\tilde{\mathbf{K}}}_t^{(l)}$ and ${\tilde{\mathbf{V}}}_t^{(l)}$ by

最后，稠密记忆网络基于 ${\tilde{\mathbf{K}}}_t^{(l)}$ 和 ${\tilde{\mathbf{V}}}_t^{(l)}$ 通过下式更新：

$$\begin{array}{}\text{(2)}& {\mathbf{M}}_{t+1}^{(l)}=\lambda {\mathbf{M}}_t^{(l)}+\frac{1-\lambda }{L_t^{\prime }}\frac{{\tilde{\mathbf{K}}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}{\tilde{\mathbf{V}}}_t^{(l)}.\end{array}$$

where $\lambda$ represents the discount factor. In addition, the query-key normalization vector can be updated by

其中 $\lambda$ 表示折扣因子。 此外，查询—键归一化向量可通过下式更新：

$$\begin{array}{}\text{(3)}& {\mathbf{S}}_{t+1}^{(l)}=\lambda {\mathbf{S}}_t^{(l)}+\frac{1-\lambda }{L_t^{\prime }}\frac{{\tilde{\mathbf{K}}}_t^{(l)\mathrm{\top }}\mathbf{1}}{\sqrt{d_k}}.\end{array}$$

This native storage procedure is presented in Figure 2(c). Based on the constructed function space, we aim to internalize various memory operations into the model's computation through optimization. Specifically, the selection and projection provide the model with compression capabilities. Meanwhile, semantic-based computation enables memory instructions to be understood and applied within the latent space.

该原生存储过程如图2(c)所示。 基于所构建的函数空间，我们旨在通过优化把各种记忆操作内化到模型计算中。 具体而言，选择和投影为模型提供了压缩能力。 同时，基于语义的计算使记忆指令能够在潜在空间中被理解和应用。

In practice, we find that replacing the linear update with a Gated Delta Network (GDN)-based update obtains better performance, so Metis finally adopts the GDN-based update (GDU) strategy. Section 6.3 and the appendix compare the two implementations through ablation studies.

在实践中，我们发现，用基于门控 Delta 网络（GDN）的更新替代线性更新可获得更好性能，因此 Metis 最终采用基于 GDN 的更新（GDU）策略。 第 6.3 节和附录通过消融研究比较这两种实现。

Native Memory Utilization Procedure. We define the memory attention as

原生记忆利用过程。 我们将记忆注意力定义为：

$$\begin{array}{}\text{(4)}& {\tilde{\mathbf{A}}}_t^{(l)}=\mathrm{diag}{\left({\tilde{\mathbf{Q}}}_t^{(l)}{\mathbf{S}}_t^{(l)}\right)}^{-1}{\tilde{\mathbf{Q}}}_t^{(l)}{\mathbf{M}}_t^{(l)}.\end{array}$$

where ${\tilde{\mathbf{Q}}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_Q^{(l)}$ denotes the memory query states with optimizable parameter ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{d\times d_k}$. In practice, we add an identity vector to the normalization denominator to prevent numerical overflow and improve numerical stability. Then, the memory attention is integrated into the main branch of attention, and replaces Equation (1) with

其中，${\tilde{\mathbf{Q}}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_Q^{(l)}$ 表示记忆查询状态，其可优化参数为 ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{d\times d_k}$。 在实践中，我们在归一化分母中加入恒等向量，以防止数值溢出并提高数值稳定性。 随后，记忆注意力被集成到注意力主分支中，并以下式替换式（1）：

$$\begin{array}{}\text{(5)}& \begin{array}{rl}{\mathbf{A}}_t^{(l)}=& \gamma \mathrm{Softmax}(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{K}}_t^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L)){\mathbf{V}}_t^{(l)}\\ & +(1-\gamma )\mathrm{Norm}\left({\tilde{\mathbf{A}}}_t^{(l)}\right).\end{array}\end{array}$$

where $\mathrm{Norm}(\cdot )$ is applied to the memory readout to align its scale with the original attention branch, ${\mathbf{K}}_t^{(l)},{\mathbf{V}}_t^{(l)}$ are input key states and value states at the current step, and $\gamma \in [0,1]$ balances the two branches.

其中，$\mathrm{Norm}(\cdot )$ 应用于记忆读出，以使其尺度与原始注意力分支对齐；${\mathbf{K}}_t^{(l)},{\mathbf{V}}_t^{(l)}$ 是当前步骤的输入键状态和值状态；$\gamma \in [0,1]$ 用于平衡两个分支。

3.4 Theoretical Insight of Native Memory Procedures

We provide theoretical insights on how information from previous steps influences subsequent inference through Metis blocks. At step $t$, we prepend an additional virtual memory prefix ${\mathbf{P}}_t^{(l)}\in {\mathbb{R}}^{L_p\times d}$ to the input ${\tilde{\mathbf{H}}}_t^{(l)}$ of the $l$-th attention layer, resulting in the augmented input

我们从理论上说明先前步骤的信息如何通过 Metis 块影响后续推理。 在步骤 $t$，我们将额外的虚拟记忆前缀 ${\mathbf{P}}_t^{(l)}\in {\mathbb{R}}^{L_p\times d}$ 添加到第 $l$ 个注意力层的输入 ${\tilde{\mathbf{H}}}_t^{(l)}$ 之前，得到增强输入：

$${\hat{\mathbf{H}}}_t^{(l)}=\left[\begin{array}{c}{\mathbf{P}}_t^{(l)}\\ {\tilde{\mathbf{H}}}_t^{(l)}\end{array}\right].$$

Then, we compute the corresponding query state ${\hat{\mathbf{Q}}}_t^{(l)}$ as follows

随后，我们按如下方式计算对应的查询状态 ${\hat{\mathbf{Q}}}_t^{(l)}$：

$${\hat{\mathbf{Q}}}_t^{(l)}={\hat{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_Q^{(l)}=\left[\begin{array}{c}{\mathbf{P}}_t^{(l)}{\mathbf{W}}_Q^{(l)}\\ {\mathbf{Q}}_t^{(l)}\end{array}\right].$$

Similarly, we have the key states and value states

类似地，我们得到键状态和值状态：

$$\begin{array}{rl}{\hat{\mathbf{K}}}_t^{(l)}& =\left[\begin{array}{c}{\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}\\ {\mathbf{K}}_t^{(l)}\end{array}\right],\\ {\hat{\mathbf{V}}}_t^{(l)}& =\left[\begin{array}{c}{\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)}\\ {\mathbf{V}}_t^{(l)}\end{array}\right].\end{array}$$

The attention output ${\hat{\mathbf{A}}}_t$ is then computed using a modified causal mask $\mathrm{Mask}(L_p+L)\in {\mathbb{R}}^{(L_p+L)\times (L_p+L)}$ by

随后，使用修改后的因果掩码 $\mathrm{Mask}(L_p+L)\in {\mathbb{R}}^{(L_p+L)\times (L_p+L)}$ 计算注意力输出 ${\hat{\mathbf{A}}}_t$：

$${\hat{\mathbf{A}}}_t=\mathrm{Softmax}(\frac{{\hat{\mathbf{Q}}}_t^{(l)}{\hat{\mathbf{K}}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L_p+L)){\hat{\mathbf{V}}}_t^{(l)}.$$

Specifically, we divide the causal mask into four parts as follows

具体而言，我们将因果掩码划分为以下四部分：

$$\mathrm{Mask}(L_p+L)=\left[\begin{array}{cc}{\mathbf{0}}_{L_p\times L_p}& -{\mathrm{\infty }}_{L_p\times L}\\ {\mathbf{0}}_{L\times L_p}& \mathrm{Mask}(L)\end{array}\right].$$

where ${\mathbf{0}}_{L\times L_p}$ allows the virtual memory prefix tokens to be visible to input tokens. Then, we decompose the calculation of attention as

其中，${\mathbf{0}}_{L\times L_p}$ 使输入 token 能够看到虚拟记忆前缀 token。 随后，我们将注意力计算分解为：

$$\begin{array}{rl}{\hat{\mathbf{A}}}_t=& \left[\begin{array}{cc}\mathrm{Softmax}\left(\frac{({\mathbf{P}}_t^{(l)}{\mathbf{W}}_Q^{(l)})({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right)& {\mathbf{0}}_{L_p\times L}\\ {\mathrm{Softmax}}^{\ast }\left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right)& {\mathrm{Softmax}}^{\ast }(\frac{{\mathbf{Q}}_t^{(l)}{\mathbf{K}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L))\end{array}\right]\\ & \cdot \left[\begin{array}{c}{\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)}\\ {\mathbf{V}}_t^{(l)}\end{array}\right].\end{array}$$

where ${\mathrm{Softmax}}^{\ast }(\cdot )$ denotes the global softmax function applied to the entire row. Then, we retain the attention outputs corresponding to the non-virtual tokens by

其中，${\mathrm{Softmax}}^{\ast }(\cdot )$ 表示应用于整行的全局 softmax 函数。 随后，我们通过下式保留与非虚拟 token 对应的注意力输出：

$$\begin{array}{}\text{(6)}& \begin{array}{rl}{\mathbf{A}}_t^{(l)}=& \underset{\text{Original Attention}}{\underbrace{{\mathrm{Softmax}}^{\ast }(\frac{{\mathbf{Q}}_t^{(l)}{\mathbf{K}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L)){\mathbf{V}}_t^{(l)}}}\\ & +\underset{\text{Memory Attention}}{\underbrace{{\mathrm{Softmax}}^{\ast }\left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right)({\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)})}}.\end{array}\end{array}$$

Let ${\mathbf{z}}_{\text{orig}}$ and ${\mathbf{z}}_{\text{mem}}$ denote the partition items of the original attention and memory attention

令 ${\mathbf{z}}_{\text{orig}}$ 和 ${\mathbf{z}}_{\text{mem}}$ 表示原始注意力与记忆注意力的配分项：

$$\begin{array}{}\text{(7)}& \begin{array}{rl}{\mathbf{z}}_{\text{orig}}& =\exp (\frac{{\mathbf{Q}}_t^{(l)}{\mathbf{K}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L)){\mathbf{1}}_L,\\ {\mathbf{z}}_{\text{mem}}& =\exp \left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right){\mathbf{1}}_{L_p}.\end{array}\end{array}$$

Then, we perform element-wise division by its element-wise sum to get the weighting matrices

随后，我们用二者逐元素之和进行逐元素除法，得到加权矩阵：

$$\begin{array}{rl}{\mathbf{\Lambda }}_{\text{orig}}& =\mathrm{diag}({\mathbf{z}}_{\text{orig}}\oslash ({\mathbf{z}}_{\text{orig}}+{\mathbf{z}}_{\text{mem}})),\\ {\mathbf{\Lambda }}_{\text{mem}}& =\mathrm{diag}({\mathbf{z}}_{\text{mem}}\oslash ({\mathbf{z}}_{\text{orig}}+{\mathbf{z}}_{\text{mem}}))=\mathbf{I}-{\mathbf{\Lambda }}_{\text{orig}}.\end{array}$$

Then, Equation (6) is equivalent to the equation with the normal Softmax function for each part:

于是，式（6）等价于对各部分使用普通 Softmax 函数的下式：

$$\begin{array}{rl}{\mathbf{A}}_t^{(l)}=& {\mathbf{\Lambda }}_{\text{orig}}\mathrm{Softmax}(\frac{{\mathbf{Q}}_t^{(l)}{\mathbf{K}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L)){\mathbf{V}}_t^{(l)}\\ & +(\mathbf{I}-{\mathbf{\Lambda }}_{\text{orig}})\mathrm{Softmax}\left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right)({\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)}).\end{array}$$

To control the influence of the two attention components, we introduce a global weighting parameter $\gamma \in [0,1]$ to approximate the original weighting matrices by

为控制两个注意力分量的影响，我们引入全局加权参数 $\gamma \in [0,1]$，通过下式近似原始加权矩阵：

$$\begin{array}{rl}{\mathbf{A}}_t^{(l)}=& \gamma \mathrm{Softmax}(\frac{{\mathbf{Q}}_t^{(l)}{\mathbf{K}}_t^{(l)\mathrm{\top }}}{\sqrt{d_k}}+\mathrm{Mask}(L)){\mathbf{V}}_t^{(l)}\\ & +(1-\gamma )\mathrm{Softmax}\left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right)({\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)}).\end{array}$$

Then, we denote this specific memory attention part for ${\mathbf{P}}_t^{(l)}$ as

随后，我们将针对 ${\mathbf{P}}_t^{(l)}$ 的这一特定记忆注意力部分表示为：

$${\check{\mathbf{A}}}_t^{(l)}=\mathrm{Softmax}\left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right)({\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)}).$$

We define the function of similarity between ${\mathbf{Q}}_t^{(l)}$ and ${\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}$ as

我们将 ${\mathbf{Q}}_t^{(l)}$ 与 ${\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}$ 之间的相似度函数定义为：

$$\mathrm{Sim}({\mathbf{Q}}_t^{(l)},{\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)})=\exp \left(\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\right).$$

Then, the memory attention can be rewritten as

于是，记忆注意力可改写为：

$$\begin{array}{rl}{\check{\mathbf{A}}}_t^{(l)}=& \mathrm{diag}{(\mathrm{Sim}({\mathbf{Q}}_t^{(l)},{\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)})\mathbf{1})}^{-1}\\ & \cdot \mathrm{Sim}({\mathbf{Q}}_t^{(l)},{\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)})({\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)}).\end{array}$$

In order to decompose the ${\mathbf{Q}}_t^{(l)}$ part and ${\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}$ part, we approximate the similarity with

为了分解 ${\mathbf{Q}}_t^{(l)}$ 部分与 ${\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}$ 部分，我们用下式近似相似度：

$$\mathrm{Sim}({\mathbf{Q}}_t^{(l)},{\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)})=\frac{{\mathbf{Q}}_t^{(l)}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}.$$

So the memory attention can be rewritten as

因此，记忆注意力可改写为：

$$\begin{array}{}\text{(8)}& \begin{array}{rl}{\check{\mathbf{A}}}_t^{(l)}=& \mathrm{diag}{\left({\mathbf{Q}}_t^{(l)}\frac{({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1}\right)}^{-1}\\ & \cdot {\mathbf{Q}}_t^{(l)}[\frac{({\mathbf{P}}_t^{(l)}{\mathbf{W}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}({\mathbf{P}}_t^{(l)}{\mathbf{W}}_V^{(l)})].\end{array}\end{array}$$

Finally, we consider the prefix tokens ${\mathbf{P}}_t^{(l)}$ as the $c$-th ($c<t$) step aggregated results ${\overline{\mathbf{H}}}_c^{(l)}$, so we get

最后，我们将前缀 token ${\mathbf{P}}_t^{(l)}$ 视为第 $c$ 个（$c<t$）步骤的聚合结果 ${\overline{\mathbf{H}}}_c^{(l)}$，从而得到：

$$\begin{array}{}\text{(9)}& \begin{array}{rl}{\check{\mathbf{A}}}_t^{(l)}=& \mathrm{diag}{\left({\mathbf{Q}}_t^{(l)}\frac{({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1}\right)}^{-1}\\ & \cdot {\mathbf{Q}}_t^{(l)}[\frac{({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_V^{(l)})].\end{array}\end{array}$$

where ${\tilde{\mathbf{W}}}_K^{(l)},{\tilde{\mathbf{W}}}_V^{(l)}$ are parameters of the hyper memory block. It is worth noting that, in practice, the reference step $c$ is not accessible in advance, and the evidence required at step $t$ may span more than a single step. Meanwhile, the memory key states and memory value states of different steps are coupled together within the fixed-size memory network ${\mathbf{M}}_t^{(l)}$ and the normalization vector ${\mathbf{S}}_t^{(l)}$, so that the non-reference steps ($j\ne c$) inevitably leak into the readout as noise.

其中，${\tilde{\mathbf{W}}}_K^{(l)},{\tilde{\mathbf{W}}}_V^{(l)}$ 是超记忆块的参数。 值得注意的是，在实践中无法预先获得参考步骤 $c$，而且步骤 $t$ 所需的证据可能跨越多个步骤。 同时，不同步骤的记忆键状态和值状态耦合在固定大小的记忆网络 ${\mathbf{M}}_t^{(l)}$ 与归一化向量 ${\mathbf{S}}_t^{(l)}$ 中，因此非参考步骤（$j\ne c$）不可避免地会以噪声形式泄漏到读出中。

To mitigate the impact of such noise, instead of directly reusing the vanilla attention query ${\mathbf{Q}}_t^{(l)}$ in Equation (9), we utilize a trainable memory query ${\tilde{\mathbf{Q}}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_Q^{(l)}$ to get

为减轻这种噪声的影响，我们不在式（9）中直接复用普通注意力查询 ${\mathbf{Q}}_t^{(l)}$，而是使用可训练的记忆查询 ${\tilde{\mathbf{Q}}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_Q^{(l)}$，得到：

$$\begin{array}{}\text{(10)}& \begin{array}{rl}{\check{\mathbf{A}}}_t^{(l)}=& \mathrm{diag}{\left({\tilde{\mathbf{Q}}}_t^{(l)}\frac{({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1}\right)}^{-1}\\ & \cdot {\tilde{\mathbf{Q}}}_t^{(l)}[\frac{({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_V^{(l)})].\end{array}\end{array}$$

where ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{d\times d_k}$ is an optimizable projection. This decouples the memory query from the original attention and offers the freedom to reshape the cross-step similarities ${\tilde{\mathbf{Q}}}_t^{(l)}{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}$. Therefore, it can emphasize the relevant steps while suppressing the irrelevant ones, thereby reducing the influence of noise when using the memory query states. We provide a detailed theoretical error analysis in Section 3.5.

其中，${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{d\times d_k}$ 是可优化投影。 这将记忆查询与原始注意力解耦，并提供重塑跨步骤相似度 ${\tilde{\mathbf{Q}}}_t^{(l)}{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}$ 的自由度。 因此，它可以突出相关步骤并抑制无关步骤，从而在使用记忆查询状态时减小噪声影响。 我们在第 3.5 节给出详细的理论误差分析。

3.5 Theoretical Error Analysis

Unlike standard Transformers that store all historical KV pairs in a growing cache, the hyper memory block compresses information into a fixed-size matrix. Suppose the model requires extracting information from the $c$-th step. The dense memory network can be expressed as

不同于在不断增长的缓存中存储全部历史 KV 对的标准 Transformer，超记忆块把信息压缩到固定大小的矩阵中。 假设模型需要提取第 $c$ 个步骤的信息。 稠密记忆网络可表示为：

$${\mathbf{M}}_t^{(l)}=\sum _{j=1}^{t-1}{\lambda }^{t-(j+1)}\frac{1-\lambda }{L_j^{\prime }}\frac{{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}}{\sqrt{d_k}}{\tilde{\mathbf{V}}}_j^{(l)}.$$

Then, we use the memory query ${\tilde{\mathbf{Q}}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_Q^{(l)}$ to extract information from the dense memory network and query-key normalization vector by

随后，我们使用记忆查询 ${\tilde{\mathbf{Q}}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_Q^{(l)}$，通过下式从稠密记忆网络和查询—键归一化向量中提取信息：

$$\begin{array}{rl}{\tilde{\mathbf{A}}}_t^{(l)}=& \mathrm{diag}{\left({\tilde{\mathbf{Q}}}_t^{(l)}{\mathbf{S}}_t^{(l)}\right)}^{-1}{\tilde{\mathbf{Q}}}_t^{(l)}{\mathbf{M}}_t^{(l)}\\ =& \mathrm{diag}{\left(\sum _{j=1}^{t-1}{\lambda }^{t-(j+1)}\frac{1-\lambda }{L_j^{\prime }}{\tilde{\mathbf{Q}}}_t^{(l)}\frac{{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1}\right)}^{-1}\\ & \cdot \left(\sum _{j=1}^{t-1}{\lambda }^{t-(j+1)}\frac{1-\lambda }{L_j^{\prime }}{\tilde{\mathbf{Q}}}_t^{(l)}\frac{{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}}{\sqrt{d_k}}{\tilde{\mathbf{V}}}_j^{(l)}\right).\end{array}$$

We define the individual terms in the above summation as

我们将上述求和中的各项定义为：

$$\begin{array}{rl}{\mathbf{U}}_j& ={\lambda }^{t-(j+1)}\frac{1-\lambda }{L_j^{\prime }}{\tilde{\mathbf{Q}}}_t^{(l)}\frac{{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1},\\ {\mathbf{R}}_{j^{\prime }}& ={\lambda }^{t-(j^{\prime }+1)}\frac{1-\lambda }{L_{j^{\prime }}^{\prime }}{\tilde{\mathbf{Q}}}_t^{(l)}\frac{{\tilde{\mathbf{K}}}_{j^{\prime }}^{(l)\mathrm{\top }}}{\sqrt{d_k}}{\tilde{\mathbf{V}}}_{j^{\prime }}^{(l)}.\end{array}$$

and ${\tilde{\mathbf{A}}}_t^{(l)}$ can be rewritten as

于是，${\tilde{\mathbf{A}}}_t^{(l)}$ 可改写为：

$${\tilde{\mathbf{A}}}_t^{(l)}=\mathrm{diag}{\left(\sum _{j=1}^{t-1}{\mathbf{U}}_j\right)}^{-1}\left(\sum _{j^{\prime }=1}^{t-1}{\mathbf{R}}_{j^{\prime }}\right).$$

Assume that our target information is stored at the $c$-step (i.e., the similarity ${\tilde{\mathbf{Q}}}_t^{(l)}{\tilde{\mathbf{K}}}_c^{(l)}$ is significantly higher than others). Then, we can further rewrite the equation as

假设目标信息存储在第 $c$ 个步骤（即相似度 ${\tilde{\mathbf{Q}}}_t^{(l)}{\tilde{\mathbf{K}}}_c^{(l)}$ 显著高于其他步骤）。 那么，我们可将该式进一步改写为：

$$\begin{array}{rl}{\tilde{\mathbf{A}}}_t^{(l)}=& \mathrm{diag}{({\mathbf{U}}_c+\sum _{\begin{array}{c}j=1\\ j\ne c\end{array}}^{t-1}{\mathbf{U}}_j)}^{-1}({\mathbf{R}}_c+\sum _{\begin{array}{c}{j}^{\prime }=1\\ j^{\prime }\ne c\end{array}}^{t-1}{\mathbf{R}}_{j^{\prime }})\\ =& {[\mathrm{diag}({\mathbf{U}}_c)(\mathbf{I}+\mathrm{diag}({\mathbf{U}}_c{)}^{-1}\mathrm{diag}\left(\sum _{\begin{array}{c}j=1\\ j\ne c\end{array}}^{t-1}{\mathbf{U}}_j\right))]}^{-1}\\ & \cdot ({\mathbf{R}}_c+\sum _{\begin{array}{c}{j}^{\prime }=1\\ j^{\prime }\ne c\end{array}}^{t-1}{\mathbf{R}}_{j^{\prime }}).\end{array}$$

According to the first-order Taylor expansion, we have

根据一阶泰勒展开，我们有：

$$\begin{array}{rl}{\tilde{\mathbf{A}}}_t^{(l)}\approx & \mathrm{diag}({\mathbf{U}}_c{)}^{-1}{\mathbf{R}}_c\\ & +\underset{{\mathit{ϵ}}_1}{\underbrace{\mathrm{diag}({\mathbf{U}}_c{)}^{-1}\sum _{\begin{array}{c}{j}^{\prime }=1\\ j^{\prime }\ne c\end{array}}^{t-1}{\mathbf{R}}_{j^{\prime }}}}\\ & -\underset{{\mathit{ϵ}}_2}{\underbrace{\mathrm{diag}({\mathbf{U}}_c{)}^{-1}\mathrm{diag}\left(\sum _{\begin{array}{c}j=1\\ j\ne c\end{array}}^{t-1}{\mathbf{U}}_j\right)\mathrm{diag}({\mathbf{U}}_c{)}^{-1}{\mathbf{R}}_c}}\\ & -\underset{{\mathit{ϵ}}_3}{\underbrace{\mathrm{diag}({\mathbf{U}}_c{)}^{-1}\mathrm{diag}\left(\sum _{\begin{array}{c}j=1\\ j\ne c\end{array}}^{t-1}{\mathbf{U}}_j\right)\mathrm{diag}({\mathbf{U}}_c{)}^{-1}\sum _{\begin{array}{c}{j}^{\prime }=1\\ j^{\prime }\ne c\end{array}}^{t-1}{\mathbf{R}}_{j^{\prime }}}}.\end{array}$$

where the first term is equivalent to ${\check{\mathbf{A}}}_t^{(l)}$ in Equation (10) with ${\tilde{\mathbf{K}}}_c^{(l)}={\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}$ and ${\tilde{\mathbf{V}}}_c^{(l)}={\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_V^{(l)}$ by

其中，当 ${\tilde{\mathbf{K}}}_c^{(l)}={\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}$ 且 ${\tilde{\mathbf{V}}}_c^{(l)}={\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_V^{(l)}$ 时，第一项通过下式等价于式（10）中的 ${\check{\mathbf{A}}}_t^{(l)}$：

$$\begin{array}{rl}\mathrm{diag}({\mathbf{U}}_c{)}^{-1}{\mathbf{R}}_c=& \mathrm{diag}{\left({\tilde{\mathbf{Q}}}_t^{(l)}\frac{{\tilde{\mathbf{K}}}_c^{(l)\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1}\right)}^{-1}{\tilde{\mathbf{Q}}}_t^{(l)}\frac{{\tilde{\mathbf{K}}}_c^{(l)\mathrm{\top }}}{\sqrt{d_k}}{\tilde{\mathbf{V}}}_c^{(l)}\\ =& \mathrm{diag}{\left({\tilde{\mathbf{Q}}}_t^{(l)}\frac{({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}\mathbf{1}\right)}^{-1}\\ & \cdot {\tilde{\mathbf{Q}}}_t^{(l)}\frac{({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_K^{(l)}{)}^{\mathrm{\top }}}{\sqrt{d_k}}({\overline{\mathbf{H}}}_c^{(l)}{\tilde{\mathbf{W}}}_V^{(l)})\\ =& {\check{\mathbf{A}}}_t^{(l)}.\end{array}$$

Therefore, there are three error terms for $\Vert {\tilde{\mathbf{A}}}_t^{(l)}-{\check{\mathbf{A}}}_t^{(l)}{\Vert }_2$. It is worth noting that ${\mathit{ϵ}}_2$ and ${\mathit{ϵ}}_3$ are structural errors caused by global normalization, whereas ${\mathit{ϵ}}_1$ is an attention error introduced by irrelevant information. Across these three terms, there is always at least one factor of ${\tilde{\mathbf{Q}}}_t^{(l)}{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}$ where $j\ne c$ in the summation. Therefore, when this similarity is low, the resulting error is expected to be small.

因此，$\Vert {\tilde{\mathbf{A}}}_t^{(l)}-{\check{\mathbf{A}}}_t^{(l)}{\Vert }_2$ 包含三个误差项。 值得注意的是，${\mathit{ϵ}}_2$ 和 ${\mathit{ϵ}}_3$ 是由全局归一化造成的结构误差，而 ${\mathit{ϵ}}_1$ 是由无关信息引入的注意力误差。 在这三个项中，求和内始终至少包含一个 $j\ne c$ 的因子 ${\tilde{\mathbf{Q}}}_t^{(l)}{\tilde{\mathbf{K}}}_j^{(l)\mathrm{\top }}$。 因此，当这一相似度较低时，产生的误差预期也较小。

3.6 Efficiency Analysis

Metis introduces native memory with limited additional inference overhead compared with external memory. The key reason is that the original attention, memory attention, and memory storage procedure can be largely executed in parallel. For the $l$-th layer at step $t$, the original attention branch computes token-token attention over the current input, while the memory utilization branch performs memory attention over the native memory state ${\mathbf{M}}_t^{(l)}$ and ${\mathbf{S}}_t^{(l)}$. These two branches depend on the same input hidden states but have no sequential dependency on each other. Therefore, memory attention does not need to wait for the output of the original attention, and its results can be fused only after both branches finish.

与外部记忆相比，Metis 以有限的额外推理开销引入原生记忆。 关键原因在于，原始注意力、记忆注意力和记忆存储过程可以在很大程度上并行执行。 对于步骤 $t$ 的第 $l$ 层，原始注意力分支在当前输入上计算 token 间注意力，而记忆利用分支在原生记忆状态 ${\mathbf{M}}_t^{(l)}$ 和 ${\mathbf{S}}_t^{(l)}$ 上执行记忆注意力。 这两个分支依赖相同的输入隐状态，但彼此不存在顺序依赖。 因此，记忆注意力不必等待原始注意力输出，只有在两个分支都完成后才需要融合结果。

The memory storage procedure can also be decoupled from the current inference path. It updates the memory state for future steps, while the current step only reads from the existing memory state. Thus, after the required hidden states are available, the storage branch can be executed in parallel with the original attention and memory utilization, instead of becoming an additional sequential stage. As a result, the layer-level latency can be expressed as

记忆存储过程也可以与当前推理路径解耦。 它为未来步骤更新记忆状态，而当前步骤只从现有记忆状态中读取。 因此，在所需隐状态可用后，存储分支可以与原始注意力和记忆利用并行执行，而不会成为额外的顺序阶段。 由此，层级时延可表示为：

$$\begin{array}{}\text{(11)}& T_{\text{parallel}}^{(l)}=max(T_{\text{orig}}^{(l)},T_{\text{util}}^{(l)},T_{\text{store}}^{(l)})+T_{\text{fuse}}^{(l)}.\end{array}$$

Moreover, Metis stores historical information in fixed-size native memory states, rather than appending retrieved textual memories to the input context. Therefore, its memory utilization cost depends mainly on the memory state size, instead of growing linearly with the number of historical interactions. This enables Metis to provide native memory capabilities while avoiding the retrieval, concatenation, and prefilling overhead commonly introduced by external memory systems.

此外，Metis 将历史信息存储在固定大小的原生记忆状态中，而不是把检索到的文本记忆追加到输入上下文。 因此，它的记忆利用成本主要取决于记忆状态大小，而不会随历史交互次数线性增长。 这使 Metis 能够提供原生记忆能力，同时避免外部记忆系统通常引入的检索、拼接和预填充开销。

4. Data Construction

In order to build Metis by mid-training based on general foundation models, we synthesize a comprehensive training dataset based on existing public datasets. This dataset consists of primary data and auxiliary data, which are used for training native memory procedures and improving generalization in complex scenarios.

为了在通用基础模型之上通过中期训练构建 Metis，我们基于现有公开数据集合成了一套综合训练数据集。 该数据集由主要数据和辅助数据组成，用于训练原生记忆过程，并提升模型在复杂场景中的泛化能力。

4.1 Primary Data

The primary data serves as the core supervision for training native memory procedures. It is designed to teach memory foundation models to perform different memory operations in the forward computation through optimization, thereby generalizing to various scenarios. The native memory procedure is acquired through optimization rather than manual rules, so the primary data must provide explicit supervision for the desired memory operations.

主要数据是训练原生记忆过程的核心监督信号。 它旨在通过优化教会记忆基础模型在前向计算中执行不同的记忆操作，从而泛化到各种场景。 原生记忆过程通过优化而非人工规则获得，因此主要数据必须为期望的记忆操作提供显式监督。

Data Principles. We highlight two data principles. First, the data should be structured as a temporally ordered sequence of interaction steps, which mirrors the time-streaming nature of online information. Second, the data should be state-consistent. The response to a later query must agree with the memory state shaped by earlier operations. Together, these two properties teach the model to store information and use it at the appropriate later step.

数据原则。 我们强调两项数据原则。 第一，数据应组织为按时间排序的交互步骤序列，以反映在线信息随时间流动的性质。 第二，数据应保持状态一致。 对后续查询的响应必须与先前操作塑造的记忆状态一致。 这两项性质共同教会模型存储信息，并在之后恰当的步骤使用它。

Instead of generating data from scratch, we synthesize the primary data from established public benchmarks. This choice offers three advantages. Mature benchmarks provide verified facts and reasoning chains, which reduce hallucination when we extend them into long interaction sequences. Their broad coverage of fiction, science, news, and logical reasoning enriches the context and improves generalization. In addition, every synthetic sample is anchored to a source fact, which keeps the corpus traceable and easy to verify.

我们没有从头生成数据，而是从成熟的公开基准中合成主要数据。 这一选择有三项优势。 成熟基准提供经过验证的事实和推理链，在我们把它们扩展为长交互序列时可以减少幻觉。 它们广泛覆盖小说、科学、新闻和逻辑推理，丰富了上下文并改善了泛化能力。 此外，每个合成样本都锚定于一个来源事实，使语料库可追溯且易于验证。

Data Summary. We select 27 public benchmarks across four memory operations, as shown in Table 1. We organize the primary data along three orthogonal dimensions. (1) Memory operation includes remember, forget, update, and reflect, which together span the core behaviors of native memory. For every operation, a structured fact serves as the unit of synthesis, and the final query is answerable only from the information introduced in the preceding turns. A remember sample states a fact and then queries it, whereas a reflect sample introduces several single-hop facts and then queries their multi-hop composition.

数据概览。 如表1所示，我们为四种记忆操作选择了 27 个公开基准。 我们沿三个相互正交的维度组织主要数据。 （1）记忆操作包括“记住”“遗忘”“更新”和“反思”，它们共同覆盖原生记忆的核心行为。 对于每种操作，一个结构化事实作为合成单元，最终查询只能依据前面轮次中引入的信息作答。 “记住”样本先陈述事实再查询，而“反思”样本会引入若干单跳事实，再查询它们的多跳组合。

An update sample modifies a previously stated fact before the query, and a forget sample revokes a previously stated fact before the query. (2) Salience of the instruction ranges from explicit memory commands to implicit statements that embed information within natural narratives. (3) Noise level, where clean sequences form the basic case and noisy sequences are produced by inserting irrelevant turns. Jointly, these dimensions encourage the memory procedure to generalize across operations, instruction styles, and noise levels.

“更新”样本在查询前修改先前陈述的事实，“遗忘”样本则在查询前撤销先前陈述的事实。 （2）指令显著性从显式记忆命令到把信息嵌入自然叙述的隐式陈述不等。 （3）噪声水平，其中干净序列构成基本情形，噪声序列则通过插入无关轮次生成。 这些维度共同促使记忆过程跨操作、指令风格和噪声水平泛化。

表1：主要数据概览。每种记忆操作的样本均由公开基准合成，并遵循不同的多轮交互骨架；最终查询的答案与前面的记忆操作保持一致。

Operation	Interaction Streaming	Source Benchmarks
Remember	Info(A₁) → Query(A)	LoCoMo, LongMemEval, NeedleInAHaystack, RULER, LongBench, ∞Bench, L-Eval, BABILong, Bamboo, NaturalQuestions, LongChat-Eval
Update	Info(A₁) → Info(A₂) → Query(A)	ZsRE, RippleEdits, KnowEdit, TemporalWiki
Forget	Info(A₁) → Info(Ā₁) → Query(A)	TOFU, WMDP, MUSE, RWKU, WhoIsHarryPotter, BLUR, LKF, CLEAR, CounterFact
Reflect	Info(A₁) → Info(B₁) → Query(A ∩ B)	MuSiQue, StrategyQA, Bamboogle

Construction Pipeline. Our data synthesis pipeline comprises three major steps, including seed extraction, static synthesis, and quality verification.

构建流程。 我们的数据合成流程包含三个主要步骤：种子提取、静态合成和质量验证。

Step 1: Seed Extraction. From each source dataset, we extract the source reference, query, and answer to form a base dialogue. Then, we summarize the underlying fact into a structured seed, which records a subject, a relation, and a target, together with operation-specific fields such as the updated target or the multi-hop chain. We also collect a pool of distractor dialogues that are logically orthogonal to each query, which are used to extend the sequence length.

步骤 1：种子提取。 我们从每个来源数据集中提取来源参考、查询和答案，以形成基础对话。 随后，我们把底层事实概括为结构化种子，记录主语、关系和目标，以及更新后目标或多跳链等操作专用字段。 我们还收集一组与各查询在逻辑上正交的干扰对话，用于扩展序列长度。

Step 2: Static Synthesis. Guided by the structured seed, a strong instruction-following language model rewrites each base dialogue into two salience styles. The explicit style phrases the reference as a clear memory instruction, while the implicit style states the same fact as a description without an explicit instruction. To cover long-range memory, we insert a variable number of distractor turns between the reference and the query, which yields the distract variant of both styles.

步骤 2：静态合成。 在结构化种子的引导下，一个强指令遵循语言模型把每段基础对话改写为两种显著性风格。 显式风格把参考表述为清晰的记忆指令，而隐式风格则以不含显式指令的描述陈述同一事实。 为覆盖长程记忆，我们在参考与查询之间插入数量可变的干扰轮次，从而得到两种风格的干扰变体。

Step 3: Quality Verification. A language model acts as an automatic judge and filters samples according to several quality criteria. The consistency check confirms that the final answer faithfully reflects the intended memory state. The orthogonality check ensures that inserted distractors do not leak the core fact, and the shortcut check removes any query that can be answered without its reference. We additionally monitor the semantic diversity of the queries to prevent template collapse. Samples that fail any check are discarded, so that only reliable samples enter the final corpus.

步骤 3：质量验证。 一个语言模型充当自动评审器，并依据若干质量标准筛选样本。 一致性检查确认最终答案忠实反映预期记忆状态。 正交性检查确保插入的干扰项不会泄露核心事实，捷径检查则移除任何无需参考即可作答的查询。 我们还监控查询的语义多样性，以防止模板坍缩。 未通过任一检查的样本都会被丢弃，从而只有可靠样本进入最终语料库。

Data Statistics. We report the statistics of the synthesized primary data in Table 2. After filtering, the corpus contains 357,137 samples and about 406 million tokens, drawn from 27 source benchmarks. The samples are distributed across explicit, implicit, and distractor styles, which balance instruction salience and noise level. The token count is dominated by the distractor samples, especially for remember, because long irrelevant contexts are inserted to strengthen long-range memory. This profile indicates that the primary data covers diverse memory operations at varied interaction lengths, which provides a solid basis for training native memory procedures.

数据统计。 我们在表2中报告合成主要数据的统计信息。 筛选后，语料库包含来自 27 个来源基准的 357,137 个样本和约 4.06 亿个 token。 样本分布在显式、隐式和干扰风格中，从而平衡指令显著性和噪声水平。 token 数主要来自干扰样本，尤其是“记住”操作，因为其中插入了较长的无关上下文以增强长程记忆。 这一分布表明，主要数据覆盖了不同交互长度下的多样记忆操作，为训练原生记忆过程提供了坚实基础。

表2：合成主要数据的统计信息。Explicit、Implicit 和 Distract 列报告各风格的样本数量，最后一列报告以百万计的 token 总数。

Operation	Sources	Explicit	Implicit	Distract	All Samples	Tokens (M)
Remember	11	14,682	13,671	28,502	56,855	362.0
Forget	9	59,900	8,251	68,120	136,271	21.7
Update	4	33,452	7,300	40,749	81,501	11.0
Reflect	3	20,646	20,615	41,249	82,510	11.4
Total	27	128,680	49,837	178,620	357,137	406.1

4.2 Auxiliary Data

The auxiliary data is used to improve the model's generalizability. It further enhances the capabilities of memory foundation models for complex scenarios, such as multi-entity tasks and mixed dialogues.

辅助数据用于提升模型的泛化能力。 它进一步增强记忆基础模型应对多实体任务和混合对话等复杂场景的能力。

Construction Principles. The primary data consists of basic memory operations interactions, ranging from single-fact operations to multi-fact reasoning cases. However, real interactions are more complex. Multiple similar facts may coexist, some facts may be revoked while others persist, and memory turns are often interleaved with ordinary conversation. The first is interference, where the model confuses similar facts or allows a forgetting operation to corrupt a retained fact in parametric spaces. The second is memory pollution, where the model applies stored values to questions that do not need them.

构建原则。 主要数据由基础记忆操作交互组成，覆盖从单事实操作到多事实推理的情形。 然而，真实交互更加复杂。 多个相似事实可能并存，部分事实可能被撤销而其他事实继续保留，记忆轮次也常与普通对话交错。 第一种问题是干扰，即模型混淆相似事实，或让遗忘操作在参数空间中破坏应保留的事实。 第二种问题是记忆污染，即模型把存储值用于不需要这些值的问题。

The auxiliary data complements the primary data by targeting exactly these scenarios. It preserves the same fact-level structure, but composes facts and dialogues into more challenging interaction patterns, which improves the generalization and robustness of native memory.

辅助数据正是通过针对这些场景来补充主要数据。 它保留相同的事实级结构，但把事实和对话组合成更具挑战性的交互模式，从而提升原生记忆的泛化能力和鲁棒性。

Data Summary. We organize the auxiliary data into four subtypes, shown in Table 3. The first two subtypes address multi-fact scenarios. (1) Multi-entity binding jointly stores two confusable facts and queries both, which trains the model to bind each value to its own fact. (2) Selective forgetting revokes one fact while the other persists, which trains the model to forget one fact selectively without collateral loss. The other two subtypes address memory pollution.

数据概览。 如表3所示，我们把辅助数据组织为四种子类型。 前两种子类型处理多事实场景。 （1）多实体绑定联合存储两个容易混淆的事实并分别查询二者，以训练模型将每个值绑定到各自的事实。 （2）选择性遗忘撤销一个事实而保留另一个事实，以训练模型有选择地遗忘一个事实而不造成附带损失。 另外两种子类型处理记忆污染。

(3) Post-memory dialogue continues an ordinary conversation right after a memory query, so the model does not carry stored values into unrelated answers. (4) Memory-irrelevant dialogue answers a question that does not need memory even when a memory state exists, so the model learns when memory should not influence the response. Therefore, these four subtypes extend the primary data to realistic mixed interactions.

（3）记忆后对话在记忆查询之后立即继续普通对话，使模型不会把存储值带入无关答案。 （4）记忆无关对话在存在记忆状态时回答一个不需要记忆的问题，使模型学会何时不应让记忆影响响应。 因此，这四种子类型把主要数据扩展到真实的混合交互场景。

表3：辅助数据概览。每种子类型把事实或对话组合为复杂交互模式，最终答案与预期记忆状态保持一致。

Auxiliary Subtype	Interaction Streaming	Construction Source
Multi-Entity Binding	Info(A₁) → Info(B₁) → Query(A) → Query(B)	Paired facts synthesized from primary source facts
Selective Forgetting	Info(A₁) → Info(B₁) → Info(B̄₁) → Query(B) → Query(A)	Paired facts synthesized from primary source facts
Post-Memory Dialogue	Info(A₁) → Query(A) → Chat	Primary memory samples with curated normal dialogues
Memory-Irrelevant Dialogue	Info(A₁) → Chat / Chat → Chat	Primary memory samples with curated normal dialogues

Construction Pipeline. The auxiliary data is built from two shared ingredients, including synthesized paired facts and prepared normal dialogues, which are then formulated into the four subtypes.

构建流程。 辅助数据由合成的成对事实和准备好的普通对话这两种共享材料构建，随后将它们组织为四种子类型。

Paired Fact Synthesis. The multi-entity binding and selective forgetting subtypes require pairs of similar facts. For each source fact, represented by a subject, relation, and value, we synthesize one confusable counterpart fact. Each counterpart is generated using one of four transformations relative to the source fact. It keeps the subject but changes the relation, keeps the relation but changes the subject, imitates the value format, or stays semantically adjacent. A language model generates each counterpart, and a verifier discards any fact that contradicts, restates, or depends on the source. We then rewrite the verified facts into natural statements, queries, and revocation snippets to ensure fluency and diversity.

成对事实合成。 多实体绑定和选择性遗忘子类型需要成对的相似事实。 对于每个由主语、关系和值表示的来源事实，我们合成一个容易混淆的对应事实。 每个对应事实都通过相对于来源事实的四种变换之一生成。 它可以保留主语但改变关系、保留关系但改变主语、模仿值的格式，或保持语义相邻。 语言模型生成每个对应事实，验证器则丢弃任何与来源事实矛盾、重复陈述或依赖来源事实的内容。 随后，我们把通过验证的事实改写为自然陈述、查询和撤销片段，以确保流畅性和多样性。

Normal Dialogue Preparation. The post-memory dialogue and memory-irrelevant dialogue subtypes require conversations that do not require access to memory. We prepare a dialogue pool from three sources. These are general assistant dialogues for everyday requests, open-domain conversations from public corpora, and entity-related dialogues that are topically related to a stored fact yet remain answerable without it. We filter out turns with memory cues, real-time facts, or unsafe content, and we remove duplicates.

普通对话准备。 记忆后对话和记忆无关对话子类型需要无需访问记忆的对话。 我们从三个来源准备对话池。 它们分别是处理日常请求的通用助手对话、来自公开语料库的开放域对话，以及在主题上与某个存储事实相关但无需该事实也能作答的实体相关对话。 我们过滤包含记忆提示、实时事实或不安全内容的轮次，并移除重复项。

Subtype Formulation. The multi-entity binding subtype states the paired facts in turn and then queries both, which forces the model to bind each value to its correct fact. The selective forgetting subtype states both facts, revokes one, and then queries both, so the revoked fact becomes unavailable while the retained fact stays correct. The post-memory dialogue subtype appends an unrelated ordinary turn after a memory query, so the model returns to normal conversation without leaking any stored value. The memory-irrelevant dialogue subtype keeps the memory state but drops its query before an ordinary turn, and it also includes standalone dialogues that carry no memory at all.

子类型构造。 多实体绑定子类型依次陈述成对事实，然后分别查询二者，迫使模型把每个值绑定到正确事实。 选择性遗忘子类型陈述两个事实、撤销其中一个，再分别查询二者，使被撤销事实不可用，而保留事实仍然正确。 记忆后对话子类型在记忆查询后追加一个无关的普通轮次，使模型回到正常对话而不泄漏任何存储值。 记忆无关对话子类型保留记忆状态，但在普通轮次前去掉相应查询，同时还包含完全不携带记忆的独立对话。

Data Statistics. We report the statistics of the synthesized auxiliary data in Table 4. The paired-fact synthesis yields 76,153 natural snippet sets after quality verification. These snippets support 76,153 multi-entity binding samples and 76,153 selective forgetting samples. The dialogue-based subtypes are larger, because they reuse the full set of primary memory samples. Post-memory dialogue contributes 357,137 samples, and memory-irrelevant dialogue contributes 100,000 samples.

数据统计。 我们在表4中报告合成辅助数据的统计信息。 经过质量验证，成对事实合成得到 76,153 组自然片段。 这些片段支持 76,153 个多实体绑定样本和 76,153 个选择性遗忘样本。 基于对话的子类型规模更大，因为它们复用了完整的主要记忆样本集合。 记忆后对话贡献 357,137 个样本，记忆无关对话贡献 100,000 个样本。

In total, the auxiliary data adds 609,443 samples that emphasize multi-fact reasoning and pollution-resistant conversation. Together with the primary data, it provides broad coverage from single-fact operations to complex mixed interactions.

辅助数据总计增加 609,443 个样本，重点训练多事实推理和抗污染对话。 它与主要数据共同提供从单事实操作到复杂混合交互的广泛覆盖。

表4：合成辅助数据的统计信息。表中报告每种子类型的样本数量，并排除所有未通过质量验证的样本。

Target	Auxiliary Subtype	Samples
Multi-fact Scenario	Multi-Entity Binding	76,153
	Selective Forgetting	76,153
Memory Pollution	Post-Memory Dialogue	357,137
	Memory-Irrelevant Dialogue	100,000
All	Total	609,443

5. Model Optimization

To empower Metis with native memory procedures, we design multiple training objectives for mid-training. These objectives primarily consist of memory reconstruction, memory operation, and regularization. The three objectives share a common likelihood form but operate on different data. They jointly shape the native memory state and procedure.

为赋予 Metis 原生记忆过程，我们为中期训练设计了多个训练目标。 这些目标主要包括记忆重构、记忆操作和正则化。 三个目标采用共同的似然形式，但作用于不同数据。 它们共同塑造原生记忆状态和过程。

5.1 Overview

We organize every training sample as a multi-step interaction $s=\{(X_t,Y_t){\}}_{t=1}^{T_s}$, following the definition in Section 2. At step $t$, the model reads the input instruction $X_t$ and generates the assistant response $Y_t=(y_{t,1},\dots ,y_{t,|Y_t|})$. All steps are forwarded sequentially, and the native memory procedure stores information from each step to the memory state before the next step starts. Therefore, the parameters ${\theta }_t$ at step $t$ already integrate the memory state from all preceding steps $\{(X_i,Y_i){\}}_{i<t}$.

按照第 2 节中的定义，我们把每个训练样本组织为多步交互 $s=\{(X_t,Y_t){\}}_{t=1}^{T_s}$。 在步骤 $t$，模型读取输入指令 $X_t$，并生成助手响应 $Y_t=(y_{t,1},\dots ,y_{t,|Y_t|})$。 所有步骤按顺序前向执行，原生记忆过程在下一步骤开始前把每个步骤的信息存入记忆状态。 因此，步骤 $t$ 的参数 ${\theta }_t$ 已经整合了所有先前步骤 $\{(X_i,Y_i){\}}_{i<t}$ 的记忆状态。

We supervise only a subset of query steps ${\mathcal{Q}}_s\subseteq \{1,\dots ,T_s\}$, where the assistant response is labeled, while the reference and operation steps remain unlabeled. However, the responses of the reference and operation steps are still generated or provided for memory state updates. In addition, all three objectives share the per-step loss below, and they differ only in how the supervised target $Y_t$ is constructed from the training data. For a supervised step $t\in {\mathcal{Q}}_s$, we define the per-step loss as the token-averaged negative log-likelihood

我们只监督查询步骤的一个子集 ${\mathcal{Q}}_s\subseteq \{1,\dots ,T_s\}$，其中助手响应带有标签，而参考步骤和操作步骤保持无标签。 不过，参考步骤和操作步骤的响应仍会被生成或提供，用于更新记忆状态。 此外，三个目标共享下面的逐步损失，区别仅在于如何从训练数据构造监督目标 $Y_t$。 对于监督步骤 $t\in {\mathcal{Q}}_s$，我们把逐步损失定义为 token 平均的负对数似然：

$$\begin{array}{}\text{(12)}& \ell (s,t)=-\frac{1}{|Y_t|}\sum _{k=1}^{|Y_t|}\log P(y_{t,k}\mid X_t,Y_{t,<k};{\theta }_t).\end{array}$$

where $Y_{t,<k}$ denotes the previously generated tokens at step $t$, and ${\theta }_t$ is conditioned on the memory state shaped by earlier steps. The loss of a sample aggregates over its supervised steps as $\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)$, which allows a single trajectory to supervise multiple responses. During mid-training, we freeze the backbone parameters and optimize only the native memory parameters.

其中，$Y_{t,<k}$ 表示步骤 $t$ 中先前生成的 token，${\theta }_t$ 以早期步骤塑造的记忆状态为条件。 一个样本的损失在其监督步骤上聚合为 $\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)$，使一条轨迹能够监督多个响应。 在中期训练期间，我们冻结骨干参数，只优化原生记忆参数。

The three objectives correspond to five data subsets, and we control their contributions through a task-weighted sampler rather than explicit loss coefficients. At training epoch $e$, the sampling probability of subset $\tau$ is

三个目标对应五个数据子集，我们通过任务加权采样器而非显式损失系数来控制其贡献。 在训练轮次 $e$，子集 $\tau$ 的采样概率为：

$$\begin{array}{}\text{(13)}& \begin{array}{rl}{\pi }_{\tau }(e)& =\frac{w_{\tau }(e)}{\sum _{{\tau }^{\prime }\in \mathcal{T}}{w}_{{\tau }^{\prime }}(e)},\\ w_{\tau }(e)& =w_{\tau }^{\mathrm{s}}+(w_{\tau }^{\mathrm{e}}-w_{\tau }^{\mathrm{s}})min(\frac{e}{E-1},1).\end{array}\end{array}$$

where $\mathcal{T}$ is the set of subsets, $E$ is the total number of epochs, and $w_{\tau }^{\text{s}},w_{\tau }^{\text{e}}$ are the start and end weights of subset $\tau$. This linear annealing forms a curriculum that gradually shifts the sampling mass from storage-oriented data toward harder long-range and regularization data. Because the weights only modulate the sampling frequency, the expected mid-training objective can be written as

其中，$\mathcal{T}$ 是子集集合，$E$ 是总训练轮数，$w_{\tau }^{\text{s}},w_{\tau }^{\text{e}}$ 是子集 $\tau$ 的起始和结束权重。 这种线性退火形成了一套课程，使采样质量逐渐从面向存储的数据转向更困难的长程数据和正则化数据。 由于权重只调节采样频率，期望的中期训练目标可写为：

$$\begin{array}{}\text{(14)}& \mathcal{L}=\sum _{\tau \in \mathcal{T}}{\pi }_{\tau }(e){\mathbb{E}}_{s\sim {\mathcal{D}}_{\tau }}[\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)].\end{array}$$

where ${\mathcal{D}}_{\tau }$ is the data of subset $\tau$, and every sampled step contributes an unweighted loss from Equation (12).

其中，${\mathcal{D}}_{\tau }$ 是子集 $\tau$ 的数据，每个采样步骤都贡献式（12）中的未加权损失。

5.2 Memory Reconstruction Objective

The memory reconstruction objective enables Metis to store and reconstruct information. It provides an important training signal during the model's warm-up phase, as initialized models typically lack such capabilities. Furthermore, it targets the upper bound of information storage, with completely lossless compression and reconstruction. However, a trade-off exists between this objective and the native memory procedure. First, reconstruction and instruction following are contradictory, as they require specificity and generalization, respectively. Second, from the perspective of prediction tasks, the native memory procedure requires lossy compression guided by input instructions. In contrast, memory reconstruction opposes lossy compression.

记忆重构目标使 Metis 能够存储和重构信息。 它在模型预热阶段提供重要训练信号，因为初始化模型通常缺乏这种能力。 此外，它以完全无损的压缩与重构为目标，逼近信息存储的上限。 不过，该目标与原生记忆过程之间存在权衡。 第一，重构和指令遵循相互矛盾，因为二者分别要求特异性和泛化能力。 第二，从预测任务角度看，原生记忆过程需要由输入指令引导的有损压缩。 相比之下，记忆重构反对有损压缩。

This objective is built on a reconstruction subset derived from the primary data in Section 4.1, denoted as ${\mathcal{D}}_{\text{rec}}$. In each sample, a reference passage is presented and stored into the memory state at an early step, and a later query step requires the model to regenerate its content. Because the supervised response $Y_t$ reproduces the stored reference, the memory state must retain the source with minimal loss. We instantiate the per-step loss over this subset as

该目标建立在第 4.1 节主要数据衍生的重构子集上，记作 ${\mathcal{D}}_{\text{rec}}$。 在每个样本中，一段参考文本在早期步骤出现并存入记忆状态，后续查询步骤要求模型重新生成其内容。 由于监督响应 $Y_t$ 复现存储的参考文本，记忆状态必须以尽可能小的损失保留来源内容。 我们在该子集上将逐步损失实例化为：

$$\begin{array}{}\text{(15)}& {\mathcal{L}}_{\text{rec}}={\pi }_{\text{rec}}(e){\mathbb{E}}_{s\sim {\mathcal{D}}_{\text{rec}}}[\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)].\end{array}$$

where the expectation averages over samples drawn from ${\mathcal{D}}_{\text{rec}}$, and $\ell (s,t)$ measures the negative log-likelihood of reconstructing the stored content at the query step $t$. By step $t$, the reference passage has already been stored in the native memory state represented within ${\theta }_t$. Minimizing ${\mathcal{L}}_{\text{rec}}$ thus drives the hyper memory block to encode the reference into a state from which the memory utilization procedure can recover it.

其中，期望在从 ${\mathcal{D}}_{\text{rec}}$ 抽取的样本上取平均，$\ell (s,t)$ 衡量在查询步骤 $t$ 重构存储内容的负对数似然。 到步骤 $t$ 时，参考文本已经存入 ${\theta }_t$ 所表示的原生记忆状态。 因此，最小化 ${\mathcal{L}}_{\text{rec}}$ 会推动超记忆块把参考文本编码为可由记忆利用过程恢复的状态。

5.3 Memory Operation Objective

While reconstruction establishes lossless storage, native memory must additionally support input-driven operations. The memory operation objective teaches Metis to remember, forget, update, and reflect, so that the memory state evolves according to the instruction at each step. It is built on the primary data, which exhibits these operations under controlled instruction salience and noise.

重构建立了无损存储，但原生记忆还必须支持由输入驱动的操作。 记忆操作目标教会 Metis 记住、遗忘、更新和反思，使记忆状态按照每个步骤的指令演化。 该目标建立在主要数据之上，其中这些操作在受控的指令显著性和噪声条件下呈现。

We use two complementary subsets of the primary data. The first subset, denoted as ${\mathcal{D}}_{\text{op}}^{\text{e/i}}$, contains the explicit and implicit samples. Explicit samples phrase the operation as a clear command, whereas implicit samples embed the same information within a natural narrative. This contrast forces the model to infer the operation from intent rather than from surface keywords. The second subset, denoted as ${\mathcal{D}}_{\text{op}}^{\text{d}}$, contains the distractor samples, where irrelevant turns are inserted between the reference and the query. It promotes long-range retention and robustness against intervening noise.

我们使用主要数据中两个互补的子集。 第一个子集记作 ${\mathcal{D}}_{\text{op}}^{\text{e/i}}$，包含显式和隐式样本。 显式样本把操作表述为清晰命令，而隐式样本把同一信息嵌入自然叙述。 这种对比迫使模型从意图而非表层关键词推断操作。 第二个子集记作 ${\mathcal{D}}_{\text{op}}^{\text{d}}$，包含在参考与查询之间插入无关轮次的干扰样本。 它促进长程保留和抵御中间噪声的鲁棒性。

In all operation samples, the supervised response stays consistent with the information from the earlier steps. For an update sample, the answer reflects the new value rather than the old one. For a forget sample, the answer no longer exposes the forgotten value. For a reflect sample, the answer composes several stored facts into multi-hop reasoning. Therefore, a single likelihood objective suffices to supervise all operations as

在所有操作样本中，监督响应都与早期步骤的信息保持一致。 对于“更新”样本，答案反映新值而不是旧值。 对于“遗忘”样本，答案不再暴露被遗忘的值。 对于“反思”样本，答案把若干存储事实组合为多跳推理。 因此，单一似然目标足以监督所有操作：

$$\begin{array}{}\text{(16)}& \begin{array}{rl}{\mathcal{L}}_{\text{op}}=& {\pi }_{\text{e/i}}(e){\mathbb{E}}_{s\sim {\mathcal{D}}_{\text{op}}^{\text{e/i}}}[\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)]\\ & +{\pi }_{\text{d}}(e){\mathbb{E}}_{s\sim {\mathcal{D}}_{\text{op}}^{\text{d}}}[\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)].\end{array}\end{array}$$

where ${\theta }_t$ now encodes the net effect of the preceding operation sequence on the memory state. Unlike reconstruction, the target is no longer a copy of the stored content, so the model learns to transform and read the memory state under the guidance of the instruction.

其中，${\theta }_t$ 此时编码先前操作序列对记忆状态的净影响。 与重构不同，目标不再是存储内容的副本，因此模型学会在指令引导下转换和读取记忆状态。

5.4 Regularization Objective

The reconstruction and operation objectives are primarily built on simple interaction patterns, which leave the model vulnerable in complex scenarios. The regularization objective mitigates two failure modes that arise when memory operates in realistic interactions. The first is interference, where similar facts are confused or a forgetting operation corrupts a retained fact. The second is memory pollution, where stored values leak into responses that do not require them. This objective is built on the auxiliary data in Section 4.2, which composes facts and dialogues into more complex and realistic interaction patterns.

重构和操作目标主要建立在简单交互模式上，这使模型在复杂场景中较为脆弱。 正则化目标缓解记忆在真实交互中运行时出现的两种失败模式。 第一种是干扰，即相似事实被混淆，或遗忘操作破坏了应保留的事实。 第二种是记忆污染，即存储值泄漏到不需要它们的响应中。 该目标建立在第 4.2 节的辅助数据之上，这些数据把事实和对话组合为更复杂、更真实的交互模式。

We use two subsets of the auxiliary data. The multi-fact subset ${\mathcal{D}}_{\text{mf}}$ targets interference. Its multi-entity binding samples jointly present two confusable facts and query the model about both, which constrains the memory utilization procedure to bind each value to its own key. Its selective forgetting samples include an instruction that revokes one fact while preserving the other, which constrains the forget operation to act locally. The memory pollution subset ${\mathcal{D}}_{\text{mp}}$ targets leakage.

我们使用辅助数据中的两个子集。 多事实子集 ${\mathcal{D}}_{\text{mf}}$ 针对干扰。 其中的多实体绑定样本联合呈现两个容易混淆的事实，并分别查询二者，从而约束记忆利用过程把每个值绑定到各自的键。 其中的选择性遗忘样本包含撤销一个事实而保留另一个事实的指令，从而约束遗忘操作只在局部生效。 记忆污染子集 ${\mathcal{D}}_{\text{mp}}$ 针对泄漏。

Its post-memory dialogue samples continue an ordinary conversation right after a memory query, and its memory-irrelevant samples answer a question that needs no memory even when a memory state exists. In both cases, they discourage the model from injecting memory into unrelated responses. These subsets act as regularization because they constrain memory behavior under more realistic and diverse interaction scenarios. The supervised targets penalize cross-fact interference, collateral forgetting, and value leakage, which suppress degenerate solutions that always read or overwrite the memory state. We define the objective as

其中的记忆后对话样本在记忆查询后立即继续普通对话，记忆无关样本则在存在记忆状态时回答一个不需要记忆的问题。 在两种情况下，它们都阻止模型把记忆注入无关响应。 这些子集通过在更真实、更多样的交互场景下约束记忆行为来发挥正则化作用。 监督目标惩罚跨事实干扰、附带遗忘和值泄漏，从而抑制总是读取或覆写记忆状态的退化解。 我们将该目标定义为：

$$\begin{array}{}\text{(17)}& \begin{array}{rl}{\mathcal{L}}_{\text{reg}}=& {\pi }_{\text{mf}}(e){\mathbb{E}}_{s\sim {\mathcal{D}}_{\text{mf}}}[\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)]\\ & +{\pi }_{\text{mp}}(e){\mathbb{E}}_{s\sim {\mathcal{D}}_{\text{mp}}}[\sum _{t\in {\mathcal{Q}}_s}\ell (s,t)].\end{array}\end{array}$$

where many samples expose multiple supervised steps, so $|{\mathcal{Q}}_s|>1$ jointly constrains the retained and the revoked facts within one interaction. For the memory-irrelevant case, the supervised step is an ordinary turn whose target is independent of the memory state.

其中，许多样本包含多个监督步骤，因此 $|{\mathcal{Q}}_s|>1$ 会在一次交互中共同约束被保留和被撤销的事实。 对于记忆无关情形，监督步骤是一个目标独立于记忆状态的普通轮次。

6. Experiments

6.1 Experimental Settings

We evaluate Metis on memory operation tasks and memory-based question-answering (QA) tasks. The memory operation task evaluates the performance of executing memory operations. In addition, to verify the effectiveness of the native memory state, we evaluate the performance on the memory-based QA task. Our major experiments focus on evaluating the native memory state and procedure primarily through relatively short-term tasks. As for the long-term capability, we explore it from the perspective of memory capability in Section 6.6.

我们在记忆操作任务和基于记忆的问答任务上评估 Metis。 记忆操作任务评估执行记忆操作的性能。 此外，为验证原生记忆状态的有效性，我们在基于记忆的问答任务上评估其性能。 我们的主要实验重点通过相对短期的任务评估原生记忆状态和记忆过程。 对于长期能力，我们在第 6.6 节从记忆容量的角度进行探索。

Datasets and Metrics. For memory operations, we employ MemOps, which is a specific benchmark focusing on memory operations, such as remembering, forgetting, and updating. In the Full setting, the model receives three complete evidence segments, containing 24 utterances. In the Gold setting, it receives only the oracle turns required for the question. We also present the performance of the Test set of our constructed dataset. For the memory-based QA task, we conduct experiments on the golden-session setting of LoCoMo (i.e., LoCoMo (Gold)), where we provide the gold evidence sessions as input.

数据集与指标。 对于记忆操作，我们采用 MemOps，这是一个专门关注记住、遗忘和更新等记忆操作的基准。 在 Full 设置中，模型接收三个完整证据片段，其中包含 24 条话语。 在 Gold 设置中，模型只接收回答问题所需的预言机轮次。 我们还给出所构建数据集的 Test 集性能。 对于基于记忆的问答任务，我们在 LoCoMo 的黄金会话设置（即 LoCoMo (Gold)）上开展实验，并把黄金证据会话作为输入。

We also utilize the contextual generation task dataset from NextMem for further analysis. This dataset evaluates whether models can utilize the provided information to answer questions correctly, consisting of SQuAD, HotpotQA, LoCoMo, and LongMemEval. In all these settings, we utilize gpt-4.1-mini to judge each prediction against its reference answer in three repeated evaluations. Then, we report the median LLM-as-a-judge score. In each dataset, we calculate the average performance (i.e., Avg.) across different types using a micro-average.

我们还使用 NextMem 的上下文生成任务数据集做进一步分析。 该数据集由 SQuAD、HotpotQA、LoCoMo 和 LongMemEval 组成，用于评估模型能否利用给定信息正确回答问题。 在所有这些设置中，我们使用 gpt-4.1-mini，在三次重复评估中逐一比较预测与参考答案。 随后，我们报告 LLM-as-a-judge 分数的中位数。 在每个数据集中，我们使用微平均计算不同类型上的平均性能（即 Avg.）。

It should be noted that, to cover a wide range of entities for memory, we extract seed entities from various public datasets to synthesize our training data, such as LoCoMo and LongMemEval. However, we do not leak their exact QA behaviors in the training phase.

需要指出的是，为覆盖广泛的记忆实体，我们从 LoCoMo 和 LongMemEval 等多个公开数据集中提取种子实体来合成训练数据。 但是，我们没有在训练阶段泄漏其具体问答行为。

Baselines. We comprehensively evaluate our approach against four categories of baselines. For backbone models evaluated with full information appended to the context, we utilize Qwen3.5 across 4B, 9B, and 27B sizes. For the partial-context baselines, we apply RAG to these backbone models. It encodes observations and queries into dense representations, and calculates the cosine similarity between queries and all observations. The top-$5$ observations are appended to the context.

基线。 我们将本方法与四类基线进行全面比较。 对于把完整信息追加到上下文中进行评估的骨干模型，我们采用 4B、9B 和 27B 三种规模的 Qwen3.5。 对于部分上下文基线，我们在这些骨干模型上应用 RAG。 它把观察和查询编码为稠密表示，并计算查询与所有观察之间的余弦相似度。 相似度最高的 $5$ 条观察会被追加到上下文中。

For TTT-based models, we evaluate Temp-LoRA as the baseline. It fuses information into the model by training a temporary LoRA module on previous text chunks during inference, encoding historical context as transient parameter updates. Specifically, we implement Temp-LoRA with corresponding sizes of Qwen3.5 backbones. Regarding parametric memory models, we compare with $\delta$-Mem, which steers attention with low-rank corrections. More details are provided in the appendix.

对于基于 TTT 的模型，我们评估 Temp-LoRA 基线。 它在推理期间基于先前文本块训练临时 LoRA 模块，把历史上下文编码为瞬时参数更新，从而将信息融合进模型。 具体而言，我们使用相应规模的 Qwen3.5 骨干实现 Temp-LoRA。 对于参数化记忆模型，我们与使用低秩修正引导注意力的 $\delta$-Mem 进行比较。 更多细节见附录。

Training Configuration. The reported Metis models are built upon Qwen3.5 backbones and trained on $8\times$ H100 GPUs. The backbone is frozen during training, and the trainable memory parameters are initialized using the key and value projection matrices of the corresponding backbone layers. We use AdamW with a learning rate of $2\times {10}^{-4}$, a constant schedule after 200 warmup steps, weight decay 0.01, $\beta =(0.9,0.999)$, $ϵ={10}^{-8}$, and gradient clipping at 1.0. Training uses BF16 and seed 42, and saves a checkpoint every 2,000 steps.

训练配置。 所报告的 Metis 模型基于 Qwen3.5 骨干构建，并在 $8\times$ H100 GPU 上训练。 训练期间骨干保持冻结，可训练记忆参数使用对应骨干层的键和值投影矩阵初始化。 我们使用 AdamW，学习率为 $2\times {10}^{-4}$，经过 200 个预热步骤后采用恒定调度，权重衰减为 0.01，$\beta =(0.9,0.999)$，$ϵ={10}^{-8}$，梯度裁剪阈值为 1.0。 训练使用 BF16 和随机种子 42，每 2,000 步保存一个检查点。

For Metis-4B, we train our model for 14,000 steps, corresponding to one epoch. For Metis-27B, we use the same number of training steps, corresponding to approximately 0.4 epochs. For Metis-9B, we use 8,000 steps (approximately 0.5728 epochs), which is selected by early stopping on validation-set performance.

对于 Metis-4B，我们训练 14,000 步，对应一个轮次。 对于 Metis-27B，我们使用相同的训练步数，对应约 0.4 个轮次。 对于 Metis-9B，我们使用 8,000 步（约 0.5728 个轮次），该步数根据验证集性能进行早停选择。

Evaluation Pipeline. For the memory operation and memory-based QA tasks, we adopt a static evaluation paradigm. Each test trajectory is divided into two sequential phases. The first phase consists of information steps, which provide the necessary context to the model. The second phase consists of query steps, where the model must answer a question based on the prior information. Finally, the evaluation calculates performance metrics by comparing the model's output in the query step with the ground truth. The prompts of the information step and query step are provided in the appendix, and the prompts of LLM-as-a-Judge are presented in the appendix.

评估流程。 对于记忆操作和基于记忆的问答任务，我们采用静态评估范式。 每条测试轨迹被划分为两个连续阶段。 第一阶段由信息步骤组成，为模型提供必要上下文。 第二阶段由查询步骤组成，模型必须根据先前信息回答问题。 最后，评估通过比较查询步骤中的模型输出与真实答案来计算性能指标。 信息步骤和查询步骤的提示见附录，LLM-as-a-Judge 的提示也在附录中给出。

6.2 Overall Performance

表5：记忆操作任务的总体性能。完整上下文和部分上下文结果以灰色显示，以在视觉上区分可访问上下文设置与无上下文比较。在无上下文设置中，最佳和次佳分数分别以粗体和下划线表示。Avg. 表示微平均性能。

Type	Method	MemOps (Gold)					Metis Test Set
		Remember	Update	Forget	Reflect	Avg.	Remember	Update	Forget	Reflect	Avg.
Full Context	Qwen3.5-4B	84.97	86.34	81.36	85.17	84.56	80.07	70.31	70.42	83.13	75.18
	Qwen3.5-9B	88.54	88.43	82.73	86.90	86.86	78.18	69.48	67.50	84.38	73.89
	Qwen3.5-27B	91.37	90.74	84.32	84.48	87.90	81.01	73.44	75.83	88.75	78.87
Partial Context	Qwen3.5-4B	38.84	33.56	24.55	21.90	30.18	70.05	63.12	59.90	67.81	64.82
	Qwen3.5-9B	30.51	26.62	20.23	11.55	22.41	70.40	55.10	55.00	64.69	60.68
	Qwen3.5-27B	37.05	35.88	22.05	16.21	28.01	70.28	63.33	66.46	60.47	65.40
No Context	Qwen3.5-4B	4.17	0.00	1.59	0.00	1.65	12.03	0.00	49.58	0.00	16.96
	Qwen3.5-9B	4.17	1.85	0.91	0.00	1.88	11.79	5.42	49.90	0.63	18.64
	Qwen3.5-27B	3.57	0.93	0.91	0.69	1.69	10.73	2.08	47.50	1.25	16.87
	Temp-LoRA-4B	15.33	10.19	2.95	4.83	8.85	15.80	27.71	15.21	17.66	19.34
	Temp-LoRA-9B	23.81	13.43	5.00	8.10	13.51	20.05	17.71	20.21	20.47	19.51
	Temp-LoRA-27B	20.68	6.48	2.50	4.83	9.70	25.94	18.65	25.21	26.87	23.86
	$\delta$-Mem	7.44	6.02	1.82	1.55	4.38	13.92	21.77	12.40	10.31	15.03
	Metis-4B	19.35	27.55	7.27	16.90	17.84	52.24	63.85	31.25	90.16	56.72
	Metis-9B	25.89	23.61	11.59	15.52	19.63	58.14	63.33	30.42	90.78	57.92
	Metis-27B	28.27	31.02	10.91	26.55	24.76	61.08	68.13	77.50	93.44	73.77

Memory Operation Tasks. The results of MemOps in the gold setting (i.e., MemOps (Gold)) and the Metis test set are presented in Table 5. Due to the page limitation, we put the experiment results and analysis of MemOps in the full setting (i.e., MemOps (Full)) in the appendix. As expected, full-context models achieve the strongest overall performance, while removing the context causes a substantial performance drop for standard backbones. Partial context preserves some information on the Metis test set but performs poorly on MemOps (Gold), showing that incomplete histories cannot reliably support memory operations. Temp-LoRA and $\delta$-Mem recover part of the lost performance, but their gains remain limited.

记忆操作任务。 MemOps 黄金设置（即 MemOps (Gold)）和 Metis 测试集的结果见表5。 受页面篇幅限制，我们把 MemOps 完整设置（即 MemOps (Full)）的实验结果和分析放在附录中。 正如预期，完整上下文模型取得最强的整体性能，而移除上下文会使标准骨干的性能大幅下降。 部分上下文在 Metis 测试集上保留了一些信息，但在 MemOps (Gold) 上表现较差，这表明不完整的历史无法可靠支持记忆操作。 Temp-LoRA 和 $\delta$-Mem 恢复了部分损失的性能，但增益仍然有限。

Under the same no-context setting, Metis achieves the best average results on both MemOps (Gold) and the Metis test set. These results suggest that Metis can preserve information in its native memory state and use it in later steps without replaying the original context. Metis-27B achieves the best average performance on both benchmarks under the no-context setting. Compared with Metis-4B and Metis-9B, it shows clear gains in remembering, updating, reflection, and overall performance. The improvement is especially large for forgetting on the Metis test set.

在相同的无上下文设置下，Metis 在 MemOps (Gold) 和 Metis 测试集上都取得最佳平均结果。 这些结果表明，Metis 可以在原生记忆状态中保存信息，并在后续步骤中使用这些信息，而无需重放原始上下文。 在无上下文设置下，Metis-27B 在两个基准上都取得最佳平均性能。 与 Metis-4B 和 Metis-9B 相比，它在记住、更新、反思和整体性能上有明显提升。 在 Metis 测试集上，遗忘操作的提升尤其显著。

These results suggest that a sufficiently large backbone can better formulate and utilize the native memory state. In addition, forgetting is still the most difficult operation on the external MemOps (Gold) benchmark, even for Metis-27B. This suggests that removing or suppressing information in a shared latent state is more difficult to generalize than storing or updating information. Overall, Metis shows strong performance on memory operation tasks in short-term scenarios.

这些结果表明，足够大的骨干能够更好地形成并利用原生记忆状态。 此外，即使对于 Metis-27B，遗忘仍然是外部 MemOps (Gold) 基准上最困难的操作。 这表明，与存储或更新信息相比，在共享潜在状态中移除或抑制信息更难泛化。 总体而言，Metis 在短期场景的记忆操作任务上表现强劲。

表6：基于记忆的问答任务总体性能。完整上下文和部分上下文结果以灰色显示。在无上下文设置中，最佳和次佳分数分别以粗体和下划线表示。Single、Multi、Temporal 和 Open 分别表示 LoCoMo (Gold) 的单跳检索、多跳检索、时间推理和开放域知识设置；短横线表示结果不适用。Avg. 表示微平均性能。

Type	Method	LoCoMo (Gold)					NextMem
		Single	Multi	Temporal	Open	Avg.	SQuAD	HotpotQA	LongMemEval	LoCoMo	Avg.
Full Context	Qwen3.5-4B	85.12	65.92	15.78	23.88	63.52	91.00	88.28	45.21	61.00	77.15
	Qwen3.5-9B	84.43	64.93	16.64	21.35	63.00	91.00	87.61	45.79	60.65	77.05
	Qwen3.5-27B	85.83	69.96	15.55	31.18	65.03	91.80	89.18	48.43	64.47	78.80
Partial Context	Qwen3.5-4B	36.28	10.79	6.17	5.62	23.54	-	-	-	-	-
	Qwen3.5-9B	34.05	8.27	7.50	2.81	21.97	-	-	-	-	-
	Qwen3.5-27B	35.63	10.07	7.73	1.97	23.17	-	-	-	-	-
No Context	Qwen3.5-4B	0.00	0.36	0.00	1.97	0.18	11.24	25.22	2.86	0.48	11.86
	Qwen3.5-9B	0.00	0.36	0.00	0.00	0.07	14.92	34.98	2.86	0.48	15.93
	Qwen3.5-27B	0.00	0.36	0.00	0.00	0.07	16.73	38.90	3.14	0.48	17.75
	Temp-LoRA-4B	10.92	11.24	1.80	26.69	9.99	26.19	38.62	9.71	11.12	24.20
	Temp-LoRA-9B	13.33	13.31	2.42	25.00	11.72	29.52	45.07	12.93	12.80	28.12
	Temp-LoRA-27B	4.29	5.49	1.25	10.67	4.24	37.68	51.46	6.57	5.86	30.97
	$\delta$-Mem	12.86	10.16	3.28	20.22	10.79	20.74	33.02	9.79	10.29	20.42
	Metis-4B	18.90	15.29	7.03	28.37	16.31	29.62	58.13	39.36	50.48	41.69
	Metis-9B	18.87	18.53	7.03	27.25	16.81	33.06	63.45	33.36	51.56	43.39
	Metis-27B	31.01	27.97	13.83	28.93	26.74	43.42	66.54	39.71	60.41	50.82

Memory-based QA Tasks. The results of the memory-based QA tasks are presented in Table 6. Full-context models provide a strong upper bound because they can directly attend to the original evidence. Their performance drops sharply when only partial context is available. Without context, the original Qwen3.5 models obtain almost zero scores on LoCoMo (Gold), confirming that the answers cannot be reliably recovered from backbone knowledge alone. In contrast, Metis achieves the best average performance on both benchmarks under the no-context setting. Metis-27B obtains the highest score in every task category, outperforming other baselines.

基于记忆的问答任务。 基于记忆的问答任务结果见表6。 完整上下文模型能够直接关注原始证据，因此提供了很强的性能上界。 当只能访问部分上下文时，其性能会急剧下降。 在没有上下文时，原始 Qwen3.5 模型在 LoCoMo (Gold) 上的分数几乎为零，这证实仅凭骨干知识无法可靠恢复答案。 相比之下，在无上下文设置下，Metis 在两个基准上都取得最佳平均性能。 Metis-27B 在每个任务类别中都取得最高分，优于其他基线。

These results show that the native memory state can preserve useful information and support question answering without replaying the original context. The advantage of Metis is especially clear on tasks with complex or long-range memory requirements. On NextMem, Metis achieves large gains on HotpotQA, LongMemEval, and LoCoMo subset. It also substantially improves multi-hop and temporal question answering on LoCoMo (Gold). It indicates that native memory remains effective for relatively simple factual questions while providing larger gains on more demanding tasks.

这些结果表明，原生记忆状态能够保存有用信息，并在不重放原始上下文的情况下支持问答。 在具有复杂或长程记忆要求的任务上，Metis 的优势尤其明显。 在 NextMem 上，Metis 在 HotpotQA、LongMemEval 和 LoCoMo 子集上取得很大增益。 它还显著提升了 LoCoMo (Gold) 上的多跳和时间问答性能。 这表明，原生记忆对于相对简单的事实问题仍然有效，同时在要求更高的任务上带来更大增益。

The strong improvement on temporal questions also suggests that a larger Metis model can better preserve and use relations across different interaction steps. However, the gain on open-domain LoCoMo (Gold) questions is relatively limited, which indicates that some task types remain difficult even with increased model capacity. In addition, we find that the improvement between Metis-4B and Metis-9B is modest, whereas Metis-27B substantially improves the average score. This pattern suggests that backbone scaling can enhance native memory capability once model capacity is reached, although the gains are not uniform across tasks.

时间问题上的显著提升还表明，更大的 Metis 模型能更好地保存和使用不同交互步骤之间的关系。 然而，LoCoMo (Gold) 开放域问题上的增益相对有限，说明即使模型容量增加，某些任务类型仍然困难。 此外，我们发现 Metis-4B 与 Metis-9B 之间的提升较小，而 Metis-27B 显著提高了平均分。 这种模式表明，一旦达到足够模型容量，扩大骨干规模可以增强原生记忆能力，尽管不同任务上的增益并不均匀。

These results indicate that Metis provides strong memory-based QA performance in both the short-term QA tasks and the relatively longer LoCoMo (Gold) setting. In the appendix, we further apply Metis to Llama and Gemma models of varying sizes. We use the same mid-training and evaluation paradigm to explore its transferability across different backbone families and scales.

这些结果表明，无论是在短期问答任务还是相对更长的 LoCoMo (Gold) 设置中，Metis 都提供了强劲的基于记忆的问答性能。 在附录中，我们进一步把 Metis 应用于不同规模的 Llama 和 Gemma 模型。 我们使用相同的中期训练与评估范式，探索其在不同骨干家族和规模之间的可迁移性。

6.3 Ablation Studies

We conduct ablation studies on Metis-4B from the perspectives of training data and model structure. Following the main experimental setup, we evaluate LoCoMo (Gold) and NextMem in memory-based QA tasks. We also use the Metis test set and MemOps (Gold) in memory operation tasks. All ablation models use Metis-4B and the same training configuration and evaluation pipeline as the main results.

我们从训练数据和模型结构两个角度对 Metis-4B 进行消融研究。 按照主要实验设置，我们在基于记忆的问答任务中评估 LoCoMo (Gold) 和 NextMem。 我们还在记忆操作任务中使用 Metis 测试集和 MemOps (Gold)。 所有消融模型均使用 Metis-4B，并采用与主结果相同的训练配置和评估流程。

Data Ablation. We evaluate the contribution of different training data through two variants. In w/o MS, we remove the Multi-fact Scenario data. In w/o MS+MP, we remove the entire auxiliary dataset, including Multi-fact and Memory Pollution data, to examine its overall contribution to memory learning and generalization. As shown in Table 7, removing the Multi-fact Scenario data consistently reduces performance across both types of tasks. This result indicates that multi-fact supervision helps Metis integrate related information and maintain a coherent memory state.

数据消融。 我们通过两个变体评估不同训练数据的贡献。 在 w/o MS 中，我们移除多事实场景数据。 在 w/o MS+MP 中，我们移除整个辅助数据集，包括多事实和记忆污染数据，以考察它对记忆学习与泛化的整体贡献。 如表7所示，移除多事实场景数据会持续降低两类任务上的性能。 这一结果表明，多事实监督有助于 Metis 整合相关信息并维持连贯的记忆状态。

The decline is more evident on MemOps (Gold) and the Metis test set, suggesting that such data is particularly important for learning reliable memory operations. Removing the entire auxiliary dataset leads to a much larger overall degradation. The drop is especially clear on the Metis test set, while performance on LoCoMo (Gold) and NextMem also decreases consistently. This shows that auxiliary data improves the robustness and generalization of native memory procedures across different scenarios.

在 MemOps (Gold) 和 Metis 测试集上，下降更加明显，说明此类数据对于学习可靠的记忆操作尤其重要。 移除整个辅助数据集会造成大得多的整体性能下降。 Metis 测试集上的下降尤其明显，而 LoCoMo (Gold) 和 NextMem 上的性能也持续降低。 这表明，辅助数据提高了原生记忆过程在不同场景中的稳健性和泛化能力。

Structure Ablation. We further ablate the main components of the native memory procedure. In w/o GDU, we replace the GDU with a linear update (LU). In w/o SA, we remove the adaptive aggregation mechanism and directly use the last-token hidden state for memory storage. In w/o OQ, we remove the optimizable memory query projection and reuse the query from the original attention. In w/o QKN, we remove query-key normalization from memory attention.

结构消融。 我们进一步消融原生记忆过程的主要组件。 在 w/o GDU 中，我们用线性更新（LU）替换 GDU。 在 w/o SA 中，我们移除自适应聚合机制，直接使用最后一个 token 的隐藏状态存储记忆。 在 w/o OQ 中，我们移除可优化的记忆查询投影，并复用原始注意力中的查询。 在 w/o QKN 中，我们从记忆注意力中移除查询-键归一化。

Among all these evaluated variants, removing adaptive aggregation causes the largest performance drop. Directly using the last token cannot effectively capture information distributed across the input sequence. As a result, the model fails to construct an informative memory state. In addition, removing query-key normalization also causes a substantial degradation, particularly on LoCoMo (Gold) and NextMem. Without this normalization, irrelevant information may introduce stronger interference.

在所有评估的变体中，移除自适应聚合导致最大的性能下降。 直接使用最后一个 token 无法有效捕获分布在输入序列中的信息。 因此，模型无法构建信息丰富的记忆状态。 此外，移除查询-键归一化也会造成显著下降，在 LoCoMo (Gold) 和 NextMem 上尤其如此。 缺少这种归一化时，无关信息可能引入更强的干扰。

Furthermore, reusing the original attention query also reduces performance across all benchmarks. The decrease is larger on the memory-based QA tasks, indicating that a separate memory query is important for distinguishing relevant historical information from noise. This observation is consistent with our theoretical analysis, where the additional query projection reshapes cross-step similarities and suppresses interference from irrelevant memory.

此外，复用原始注意力查询也会降低所有基准上的性能。 基于记忆的问答任务下降幅更大，表明独立的记忆查询对于区分相关历史信息与噪声很重要。 这一观察与我们的理论分析一致，其中额外的查询投影会重塑跨步骤相似性，并抑制无关记忆的干扰。

We also find that replacing the GDU with a linear update has a small effect on the overall average. The linear update performs slightly better on MemOps (Gold) and the Metis test set but is clearly weaker on LoCoMo (Gold). This suggests that a linear update can handle simple and short-term memory operations, while the GDU provides a better balance in long-term scenarios. This result is consistent with our engineering observation that GDU may produce more stable model behavior, motivating its use in Metis. Appendix further compares LU and GDU across model scales, while Appendix repeats the data and structure ablations from the LU baseline.

我们还发现，用线性更新替换 GDU 对整体平均值影响很小。 线性更新在 MemOps (Gold) 和 Metis 测试集上略好，但在 LoCoMo (Gold) 上明显较弱。 这表明线性更新可以处理简单和短期的记忆操作，而 GDU 在长期场景中提供了更好的平衡。 这一结果与我们的工程观察一致，即 GDU 可能产生更稳定的模型行为，这也是 Metis 采用它的原因。 附录进一步比较不同模型规模下的 LU 与 GDU，而另一项附录实验则从 LU 基线重复数据和结构消融。

表7：Metis-4B 的消融研究结果。Avg. 表示不同组中的宏平均性能，ΔAvg. 表示与完整模型相比的相对性能差距。

Type	Model	Memory Operation Task				Memory-based QA Task				Overall
		MemOps (Gold)	Metis Test Set	Avg.	$\Delta$Avg.	LoCoMo (Gold)	NextMem	Avg.	$\Delta$Avg.	Avg.	$\Delta$Avg.
Full Model	Metis	17.84	56.72	37.28	-	16.31	41.69	29.00	-	33.14	-
Data Ablation	w/o MS	14.64	51.53	33.08	-11.26%	14.78	37.79	26.29	-9.36%	29.68	-10.43%
	w/o MS+MP	14.45	42.17	28.31	-24.07%	14.18	36.17	25.17	-13.20%	26.74	-19.31%
Structure Ablation	w/o GDU	18.50	58.54	38.52	3.32%	11.97	42.78	27.37	-5.60%	32.95	-0.58%
	w/o SA	3.67	19.72	11.70	-68.63%	9.84	18.49	14.16	-51.16%	12.93	-60.98%
	w/o OQ	13.89	53.93	33.91	-9.04%	11.46	37.07	24.26	-16.33%	29.09	-12.23%
	w/o QKN	9.32	48.74	29.03	-22.13%	10.00	26.80	18.40	-36.55%	23.72	-28.44%

6.4 Out-of-Distribution Memory Tasks

To examine whether the strong performance reported in Section 6.2 generalizes beyond the data-construction distribution, we further evaluate Metis on two out-of-distribution (OOD) benchmarks that were not used to construct the training data. All methods are evaluated under the no-context setting. We evaluate ATM-Bench on its official standard split. For MemDaily, we use the subset of the official pre-generated release in which the annotated retrieval-target messages occur before the query.

为考察第 6.2 节报告的强劲性能能否泛化到数据构建分布之外，我们进一步在两个未用于构建训练数据的分布外基准上评估 Metis。 所有方法均在无上下文设置下评估。 我们在 ATM-Bench 的官方标准划分上进行评估。 对于 MemDaily，我们使用官方预生成版本中带标注的检索目标消息出现在查询之前的子集。

In the Gold setting, the model receives only benchmark-annotated evidence: human-annotated memory items represented as text in SGM for ATM-Bench, and retrieval-target messages for MemDaily. ATM-Bench scores list-recall, number, and open-ended questions using Jaccard similarity, post-processed exact match, and an LLM judge, respectively. MemDaily reports deterministic single-choice accuracy for all six question types.

在 Gold 设置中，模型只接收基准标注的证据：ATM-Bench 中以 SGM 文本表示的人工标注记忆项，以及 MemDaily 中的检索目标消息。 ATM-Bench 分别使用 Jaccard 相似度、后处理精确匹配和 LLM 评判器，对列表召回、数字和开放式问题评分。 MemDaily 报告全部六种问题类型上的确定性单项选择准确率。

表8：分布外记忆基准上的结果。最佳和唯一的次佳分数分别以粗体和下划线表示。平均分按照官方类别数量计算。

Method	ATM-Bench (Gold)				MemDaily (Gold)
	List	Number	Open	Avg.	Aggreg.	Comp.	Cond.	Noisy	Post-proc.	Simple	Avg.
$\delta$-Mem	0.00	1.94	3.11	2.27	29.44	21.14	44.40	38.40	59.00	45.58	39.84
Temp-LoRA-4B	0.00	0.00	5.06	2.57	30.74	30.49	50.80	43.60	60.60	52.01	44.92
Temp-LoRA-9B	0.00	0.00	5.64	2.86	45.24	31.91	58.40	46.40	66.80	59.04	51.42
Temp-LoRA-27B	0.00	0.00	5.06	2.57	61.90	40.24	61.00	50.60	74.20	68.67	59.45
Metis-4B	1.08	14.17	9.92	10.22	45.02	54.67	51.00	44.60	64.80	54.62	52.54
Metis-9B	0.00	24.72	15.18	16.49	30.30	34.35	53.00	43.60	66.80	54.22	47.29
Metis-27B	0.00	31.39	14.59	18.56	40.69	66.06	56.60	52.80	75.40	61.45	59.04

As shown in Table 8, Metis demonstrates strong OOD transfer on ATM-Bench, consistently outperforming the memory baselines across model scales and most question types. The advantage also holds for the deterministically scored number questions, indicating that the improvement is not merely an artifact of the LLM judge used for open-ended questions. Since ATM-Bench requires models to retain and integrate heterogeneous evidence extracted from personal archives, these results suggest that the native memory procedure learned by Metis transfers beyond the patterns observed during training. The results on MemDaily are more mixed: Metis remains competitive but does not consistently lead the memory baselines. Together, the two benchmarks provide evidence that Metis's native memory capability generalizes to benchmarks not used in constructing its training data.

如表8所示，Metis 在 ATM-Bench 上表现出很强的分布外迁移能力，在不同模型规模和大多数问题类型上持续优于记忆基线。 这种优势在采用确定性评分的数字问题上同样成立，说明提升并非仅仅源自开放式问题使用的 LLM 评判器。 由于 ATM-Bench 要求模型保留并整合从个人档案中提取的异构证据，这些结果表明 Metis 学到的原生记忆过程能够迁移到训练期间所见模式之外。 MemDaily 上的结果更加复杂：Metis 仍具竞争力，但没有持续领先记忆基线。 综合来看，这两个基准证明了 Metis 的原生记忆能力可以泛化到未用于构建训练数据的基准。

6.5 Source-Exclusion Study

Complementing the OOD evaluation, we study how sensitive Metis is to the composition of the public sources used by the synthesis pipeline. Specifically, we remove all training instances generated from LoCoMo and LongMemEval, and train Metis-4B, Metis-9B, and Metis-27B on the remaining data. We keep other training and evaluation configurations unchanged. The evaluation is memory-only, without replaying the original context. Under the accounting used for this experiment, the exclusion removes only about 2.61% of training instances, but these removals are concentrated in the remember, reconstruction, and multi-entity or mixed-operation slices.

作为分布外评估的补充，我们研究 Metis 对合成流程所用公开来源组成的敏感程度。 具体而言，我们移除所有由 LoCoMo 和 LongMemEval 生成的训练实例，并在剩余数据上训练 Metis-4B、Metis-9B 和 Metis-27B。 其他训练和评估配置保持不变。 该评估仅使用记忆，不重放原始上下文。 按照本实验的统计口径，排除操作只移除了约 2.61% 的训练实例，但这些实例集中在记住、重构以及多实体或混合操作切片中。

表9：来源排除研究结果。括号中给出相对于 Metis 结果的百分点差值；正差值加粗，负差值以灰色显示。6-Bench Avg 是 LoCoMo、NextMem Avg、Metis Test、MemOps、ATM 和 MemDaily 的非加权平均值。

Model	LoCoMo	NextMem			Metis Test	MemOps	ATM	MemDaily	6-Bench Avg
		LongMemEval	LoCoMo	Avg.
Metis-4B	13.18 (-3.13)	29.71 (-9.65)	42.46 (-8.02)	37.30 (-4.39)	63.26 (+6.54)	16.24 (-1.60)	12.54 (+2.32)	42.85 (-9.69)	30.90 (-1.66)
Metis-9B	15.72 (-1.09)	32.00 (-1.36)	43.06 (-8.50)	41.54 (-1.85)	66.93 (+9.01)	19.30 (-0.33)	11.65 (-4.84)	41.84 (-5.45)	32.83 (-0.76)
Metis-27B	21.15 (-5.59)	44.21 (+4.50)	52.03 (-8.38)	53.31 (+2.49)	67.66 (-6.11)	30.89 (+6.13)	12.44 (-6.12)	52.10 (-6.94)	39.59 (-2.69)

As shown in Table 9, the six-benchmark macro-average decreases at all three model sizes, but the reductions remain limited rather than producing a capability cliff. The LoCoMo overall score and the LoCoMo subset of NextMem both decrease across all sizes, indicating measurable sensitivity to source composition. However, LongMemEval and the NextMem average do not follow the same pattern, and both of them show improvement for Metis-27B. A related contrast appears between the Metis test set and MemOps. They move in opposite directions, with the direction of the contrast reversing across model sizes.

如表9所示，三个模型规模上的六基准宏平均值均有所下降，但下降幅度有限，并未出现能力断崖。 LoCoMo 总体分数和 NextMem 的 LoCoMo 子集在所有规模上都下降，说明模型对来源组成存在可测量的敏感性。 然而，LongMemEval 和 NextMem 平均值没有遵循同一模式，并且二者在 Metis-27B 上都得到提升。 Metis 测试集与 MemOps 之间也出现了相关的对比。 二者朝相反方向变化，而且这种对比的方向会随模型规模反转。

Taken together, these results suggest that excluding these sources preserves most of the overall memory capability while redistributing performance across benchmarks, rather than causing uniform degradation or improvement. In addition, the average across the two OOD benchmarks, ATM and MemDaily, decreases at every model size, although the two benchmarks do not change uniformly.

综合来看，这些结果表明，排除这些来源会保留大部分整体记忆能力，同时在不同基准之间重新分配性能，而不是造成一致的下降或提升。 此外，ATM 和 MemDaily 两个分布外基准的平均值在每个模型规模上都下降，尽管两个基准的变化并不一致。

These changes may be explained from two perspectives. From the perspective of memory as a prediction problem, narrower entity coverage may make the model less sensitive to information involving unseen or low-frequency entities and to how that information may be used in future interactions. From the training-data perspective, a small but concentrated exclusion can produce a disproportionate change in task proportions, shifting the relative supervision across memory behaviors and potentially producing different performance trade-offs across benchmarks.

这些变化可以从两个角度解释。 从把记忆视为预测问题的角度看，更窄的实体覆盖可能使模型对涉及未见或低频实体的信息，以及这些信息在未来交互中的使用方式不够敏感。 从训练数据角度看，规模很小但高度集中的排除会不成比例地改变任务占比，转移不同记忆行为之间的相对监督，并可能在各基准间产生不同的性能权衡。

6.6 Memory Capacity Studies

In this part, we further explore the long-term memory capability of Metis. We evaluate this capability by modeling memory capacity, which comprises step-level and trajectory-level capacity. Specifically, step-level capacity refers to the maximum number of tokens accommodated within a single update. Similarly, trajectory-level capacity denotes the maximum number of update steps within an interaction trajectory.

在这一部分，我们进一步探索 Metis 的长期记忆能力。 我们通过对记忆容量建模来评估该能力，记忆容量包括步骤级容量和轨迹级容量。 具体而言，步骤级容量指单次更新可容纳的最大 token 数量。 类似地，轨迹级容量表示一条交互轨迹中的最大更新步骤数。

We construct a testing dataset, which contains 20 fictional users defined over a shared schema of 40 distinct and atomic persona domains. First, the model generates 40 fine-grained domains such as demographics, education and relationships. Then, for each user, the domains are shuffled with a fixed seed and instantiated sequentially. Previously generated attributes are provided as an immutable context to ensure logical consistency within each persona. Each attribute is initially expanded into a biography-style sentence conditioned on the complete persona.

我们构建了一个测试数据集，其中包含 20 个虚构用户，他们基于由 40 个不同且原子化的人格领域组成的共享模式定义。 首先，模型生成 40 个细粒度领域，例如人口统计、教育和人际关系。 随后，对于每个用户，使用固定种子打乱这些领域并依次实例化。 先前生成的属性作为不可变上下文提供，以确保每个人格内部的逻辑一致性。 每个属性最初都基于完整人格扩展为一句传记风格的句子。

After that, one direct question is generated for each domain and reused across all users. Finally, each biography-style message is rewritten as a concise first-person statement that expresses only the corresponding fact. During evaluation, the 40 records of each user form an ordered trajectory. The statements are sequentially stored in memory, and the queries are used to probe the model. In addition, the user attributes serve as the gold answer for LLM-based judging. Based on this dataset, we compare the performance of Qwen3.5-4B with full context and Metis-4B under two evaluation settings.

之后，每个领域生成一个直接问题，并在所有用户之间复用。 最后，每条传记风格消息都被改写为只表达相应事实的简洁第一人称陈述。 评估期间，每个用户的 40 条记录构成一条有序轨迹。 这些陈述被依次存入记忆，查询则用于探测模型。 此外，用户属性作为基于 LLM 评判的黄金答案。 基于该数据集，我们在两种评估设置下比较使用完整上下文的 Qwen3.5-4B 与 Metis-4B 的性能。

图3：步骤级（左）和轨迹级（右）记忆容量结果。

Step-level Capacity. This setting evaluates how much information the model can encode within a single memory update. For each user of step $t$, the memory state is reset, and the first $t$ statements are concatenated and updated in one operation. The model is then queried about the first, middle, and last facts in the updated content. Since the full history is encoded from scratch at each step, this setting isolates the capacity of a single update as the input length increases. The results are presented in Figure 3 (left).

步骤级容量。 该设置评估模型在单次记忆更新中可以编码多少信息。 对于步骤 $t$ 中的每个用户，记忆状态会被重置，并将前 $t$ 条陈述连接起来，在一次操作中完成更新。 随后，向模型查询更新内容中的第一条、中间一条和最后一条事实。 由于每一步都从头编码完整历史，该设置隔离了输入长度增加时单次更新的容量。 结果见图3（左）。

For the step-level setting, Metis performs well when a single update contains only a small amount of information, but its accuracy decreases rapidly as the input becomes longer. The performance on the first fact shows the clearest downward trend, while the middle and last facts exhibit larger fluctuations. When the input exceeds several hundred words, performance at all three positions becomes low. In contrast, the full-context baseline remains much stronger, especially for the first fact.

在步骤级设置中，当单次更新仅包含少量信息时，Metis 表现良好，但随着输入变长，其准确率会迅速下降。 第一条事实的性能呈现最清晰的下降趋势，而中间和最后一条事实的波动更大。 当输入超过数百个词时，三个位置的性能都变得较低。 相比之下，完整上下文基线仍然强得多，尤其是在第一条事实上。

Trajectory-level Capacity. This setting evaluates how much information the model can retain over a sequence of memory updates. For each user, the memory state is reset only once and then accumulates throughout the trajectory. The 40 statements are divided into consecutive groups of $g$ statements, with $g=5$ by default, and each group is concatenated and updated in Metis sequentially. After every group-level update, the model is queried about the first, middle, and current updated facts. This setting measures the capacity of the evolving memory state under repeated updates, with performance reported against the number of updated trajectory steps. The results are presented in Figure 3 (right).

轨迹级容量。 该设置评估模型在一系列记忆更新过程中可以保留多少信息。 对于每个用户，记忆状态只重置一次，随后在整条轨迹中持续累积。 40 条陈述被划分为若干连续的组，每组包含 $g$ 条陈述，默认 $g=5$；各组分别连接后依次更新进 Metis。 每次组级更新之后，向模型查询第一条、中间一条和当前更新的事实。 该设置衡量重复更新下演化记忆状态的容量，并依据已更新的轨迹步骤数报告性能。 结果见图3（右）。

For the trajectory-level setting, performance also declines as the number of updated steps increases. The accuracy of the first fact decreases almost continuously, showing that early information is gradually weakened by later updates. The middle and most recent facts also remain unstable, which suggests that new updates introduce interference throughout the whole memory state rather than only overwriting the oldest information. Although the amount of information in each update is fixed, performance drops clearly as the trajectory becomes longer. This confirms that repeated state transitions and accumulated compression errors form another major limitation of native memory.

在轨迹级设置中，性能也会随更新步骤数增加而下降。 第一条事实的准确率几乎持续降低，表明早期信息会被后续更新逐渐削弱。 中间和最近事实的表现也不稳定，这表明新更新会在整个记忆状态中引入干扰，而不只是覆写最旧的信息。 尽管每次更新中的信息量固定，随着轨迹变长，性能仍然明显下降。 这证实，重复状态转移和累积压缩误差构成了原生记忆的另一项主要局限。

6.7 General Capability Studies

Previous experiments have demonstrated the effectiveness of Metis on memory-related tasks. However, integrating native memory into the forward computation potentially influences the backbone's original behavior, which possibly decreases its general capabilities. In this part, we further explore how Metis performs on the general tasks compared with its original backbone. We compare Metis-4B with Qwen3.5-4B and report the performance difference between them. Specifically, we design two settings to evaluate the general capabilities of Metis in different stages.

先前实验已经证明 Metis 在记忆相关任务上的有效性。 然而，把原生记忆集成到前向计算中可能会影响骨干的原始行为，进而降低其通用能力。 在这一部分，我们进一步探索 Metis 与原始骨干相比在通用任务上的表现。 我们比较 Metis-4B 与 Qwen3.5-4B，并报告二者之间的性能差异。 具体而言，我们设计了两种设置，用于评估 Metis 在不同阶段的通用能力。

The first is the Initial Stage, which measures performance on general tasks before any information is stored in memory, corresponding to step $t=1$. In this setting, Metis is reset to an empty memory state before receiving the original prompt in benchmarks, and the backbone receives the same prompt. The second is the Active Stage, which evaluates Metis after it has accumulated irrelevant information over previous interaction steps, where we have the step $t>1$. For Metis, we reset the memory state and store task-irrelevant messages before providing the benchmark prompt. For the backbone, we prepend the same messages to its prompt.

第一种是初始阶段，用于衡量在记忆中尚未存入任何信息之前，即步骤 $t=1$ 时，模型在通用任务上的性能。 在该设置中，Metis 在接收基准原始提示前被重置为空记忆状态，骨干则接收相同提示。 第二种是激活阶段，用于评估 Metis 在先前交互步骤中累积无关信息之后的表现，此时步骤 $t>1$。 对于 Metis，我们重置记忆状态，并在提供基准提示前存储与任务无关的消息。 对于骨干，我们把相同消息前置到它的提示中。

Our experiments are conducted under MMLU-Pro, IFEval, GSM8K, and MMMLU. In IFEval, we adopt the strict evaluation setting, which verifies instruction compliance directly on the original model response without applying the response transformations used by the loose criterion. The detailed prompts of irrelevant messages are provided in the appendix. We present the results in Table 10.

我们的实验在 MMLU-Pro、IFEval、GSM8K 和 MMMLU 上进行。 在 IFEval 中，我们采用严格评估设置，直接在原始模型响应上验证指令遵循情况，不应用宽松标准所使用的响应转换。 无关消息的详细提示见附录。 结果见表10。

表10：通用能力任务结果。Gap 计算为 Metis-4B 的性能减去 Qwen3.5-4B 的性能。

Benchmark	Initial Stage			Active Stage
	Qwen3.5-4B	Metis-4B	Gap	Qwen3.5-4B	Metis-4B	Gap
MMLU-Pro	46.00	45.20	-0.80	46.00	40.90	-5.10
IFEval	79.30	79.85	+0.55	76.71	54.53	-22.18
GSM8K	83.09	82.03	-1.06	84.53	78.92	-5.61
MMMLU	61.30	60.50	-0.80	59.90	56.60	-3.30

The results show that Metis largely preserves the general capabilities of its original backbone at the initial stage. It shows only minor decreases on the other tasks. This indicates that the added memory architecture and memory-specific training do not substantially change the model's behavior when the memory state is empty. A different trend appears at the active stage.

结果表明，在初始阶段，Metis 基本保留了原始骨干的通用能力。 它在其他任务上仅出现小幅下降。 这说明，当记忆状态为空时，新增的记忆架构和记忆专项训练不会实质性改变模型行为。 在激活阶段则出现了不同趋势。

After irrelevant information is stored, Metis shows consistent performance drops across all benchmarks. The degradation is moderate on MMLU-Pro, GSM8K, and MMMLU, but is much larger on IFEval. This suggests that irrelevant native memory may introduce noise into the forward computation and interfere with the processing of the current general task, especially on strict instruction following. Overall, Metis retains most of its original general capability before memory is activated, but drops as more information is stored in the memory states.

存储无关信息后，Metis 在所有基准上都持续出现性能下降。 MMLU-Pro、GSM8K 和 MMMLU 上的下降较为温和，但 IFEval 上的下降大得多。 这表明，无关的原生记忆可能把噪声引入前向计算并干扰当前通用任务的处理，尤其会影响严格的指令遵循。 总体而言，在记忆激活之前，Metis 保留了大部分原始通用能力，但随着记忆状态中存储更多信息，其性能会下降。

6.8 Low-rank Decomposition

Storage overhead is a key efficiency metric for memory. In short-term tasks with limited information, parametric memory representations can be further compressed. Therefore, to explore the storage optimization potential of Metis, we apply low-rank decomposition to the memory states for efficient storage and reconstruct them before memory utilization. Specifically, we cast the memory state ${\mathbf{M}}_t^{(l)}$ to FP32 and compute an SVD along its last two dimensions. Therefore, for the retained rank $k$, the decomposition process can be represented as

存储开销是记忆的一项关键效率指标。 在信息量有限的短期任务中，参数化记忆表示可以进一步压缩。 因此，为探索 Metis 的存储优化潜力，我们对记忆状态应用低秩分解以实现高效存储，并在利用记忆之前对其进行重构。 具体而言，我们把记忆状态 ${\mathbf{M}}_t^{(l)}$ 转换为 FP32，并沿其最后两个维度计算 SVD。 因此，对于保留秩 $k$，分解过程可表示为：

$${\mathbf{M}}_t^{(l)}={\mathbf{U}}_t^{(l)}{\mathbf{\Sigma }}_t^{(l)}{\mathbf{V}}_t^{(l)\mathrm{\top }}.$$

We maintain the low-rank approximation instead of the original full-rank factors.

我们保留下列低秩近似，而不是原始的满秩因子：

$$\begin{array}{rl}{\hat{\mathbf{U}}}_t^{(l)}& ={\mathbf{U}}_{t,:,1:k}^{(l)},\\ {\hat{\mathbf{\Sigma }}}_t^{(l)}& ={\mathbf{\Sigma }}_{t,1:k,1:k}^{(l)},\\ {\hat{\mathbf{V}}}_t^{(l)}& ={\mathbf{V}}_{t,:,1:k}^{(l)}.\end{array}$$

Here, ${\mathbf{U}}_{t,:,1:k}^{(l)}$ and ${\mathbf{V}}_{t,:,1:k}^{(l)}$ denote the first $k$ columns of ${\mathbf{U}}_t^{(l)}$ and ${\mathbf{V}}_t^{(l)}$, respectively, while ${\mathbf{\Sigma }}_{t,1:k,1:k}^{(l)}$ denotes the leading $k\times k$ diagonal submatrix of ${\mathbf{\Sigma }}_t^{(l)}$. Then, the reconstruction can be expressed as

其中，${\mathbf{U}}_{t,:,1:k}^{(l)}$ 和 ${\mathbf{V}}_{t,:,1:k}^{(l)}$ 分别表示 ${\mathbf{U}}_t^{(l)}$ 和 ${\mathbf{V}}_t^{(l)}$ 的前 $k$ 列，而 ${\mathbf{\Sigma }}_{t,1:k,1:k}^{(l)}$ 表示 ${\mathbf{\Sigma }}_t^{(l)}$ 左上角的 $k\times k$ 对角子矩阵。 随后，重构可表示为：

$${\hat{\mathbf{M}}}_t^{(l)}={\hat{\mathbf{U}}}_t^{(l)}{\hat{\mathbf{\Sigma }}}_t^{(l)}{\hat{\mathbf{V}}}_t^{(l)\mathrm{\top }}.$$

We evaluate $k\in \{1,4,16,64,128,256\}$ on the four benchmarks with Metis-4B aligned with Section 6.2. We compare these models with low-rank memory states to the original Metis models to characterize the trends of their performance degradation, where the full memory-state dimension is 1024. The results are presented in Figure 4, which shows a clear trend as the retained rank increases.

我们使用与第 6.2 节一致的 Metis-4B，在四个基准上评估 $k\in \{1,4,16,64,128,256\}$。 我们将采用低秩记忆状态的模型与原始 Metis 模型进行比较，以刻画其性能下降趋势，其中完整记忆状态维度为 1024。 结果见图4，随着保留秩增大，图中呈现出清晰趋势。

图4：不同记忆状态秩下的低秩分解结果。

Extremely low ranks, such as 1 and 4, cause substantial performance degradation. This indicates that a few singular directions are insufficient to preserve the semantic information stored in the memory states. Performance improves rapidly when the rank increases to 16 and becomes close to the full-rank model at rank 64. Further increasing the rank from 64 to 256 brings almost no additional improvement. These results suggest that useful information in the memory states is mainly concentrated in a relatively low-dimensional subspace.

秩极低时，例如 1 和 4，会导致显著的性能下降。 这表明，少数几个奇异方向不足以保存记忆状态中存储的语义信息。 当秩增至 16 时，性能快速提升；在秩为 64 时，性能已接近满秩模型。 把秩从 64 进一步增加到 256 几乎不会带来额外提升。 这些结果表明，记忆状态中的有用信息主要集中在一个相对低维的子空间中。

表11：不同数据集上的低秩分解结果。括号中的值表示相对于 Full 的恢复比例；高于 100% 的值被视为评判器波动，而不是性能提升。

Dataset	$k=1$	$k=4$	$k=16$	$k=64$	$k=128$	$k=256$	Full
LoCoMo (Gold)	11.31 (69.4%)	10.81 (66.3%)	14.21 (87.1%)	16.00 (98.1%)	16.36 (100.3%)	16.14 (99.0%)	16.31 (100.0%)
NextMem	22.80 (54.7%)	27.83 (66.7%)	38.69 (92.8%)	41.43 (99.4%)	41.78 (100.2%)	41.71 (100.0%)	41.69 (100.0%)
Metis Test	17.99 (31.7%)	41.26 (72.7%)	53.64 (94.6%)	56.63 (99.8%)	56.75 (100.1%)	56.04 (98.8%)	56.72 (100.0%)
MemOps (Gold)	5.60 (31.4%)	11.49 (64.4%)	18.36 (102.9%)	18.36 (102.9%)	18.17 (101.8%)	18.79 (105.3%)	17.84 (100.0%)
Overall	14.43 (43.5%)	22.84 (68.9%)	31.22 (94.2%)	33.10 (99.9%)	33.26 (100.4%)	33.17 (100.1%)	33.14 (100.0%)

Table 11 further shows the sensitivity to low-rank decomposition in different datasets. The results show that different benchmarks have different levels of sensitivity to aggressive low-rank compression. At very small ranks, performance drops clearly on all datasets. The decrease is particularly large on the Metis test set and MemOps (Gold), suggesting that memory operation tasks require sufficient representational capacity to preserve operation-related information.

表11进一步展示了不同数据集对低秩分解的敏感性。 结果表明，不同基准对激进低秩压缩的敏感程度不同。 当秩非常小时，所有数据集上的性能都明显下降。 Metis 测试集和 MemOps (Gold) 上的下降尤其明显，说明记忆操作任务需要足够的表示容量来保存与操作相关的信息。

LoCoMo (Gold) is relatively less sensitive at $k=1$, although its performance fluctuates at very small ranks. This may indicate that some useful conversational information is concentrated in a small number of dominant directions. However, these directions alone are not sufficient to support stable memory utilization. MemOps (Gold) recovers its full-rank performance at $k=16$ and remains stable at larger ranks. In contrast, the Metis test set continues to improve from $k=16$ to $k=64$. This difference suggests that the two memory operation benchmarks require different levels of memory capacity.

LoCoMo (Gold) 在 $k=1$ 时相对不那么敏感，尽管其性能在非常小的秩下会波动。 这可能表明，一些有用的对话信息集中在少数几个主导方向中。 然而，仅靠这些方向不足以支持稳定的记忆利用。 MemOps (Gold) 在 $k=16$ 时恢复满秩性能，并在更大的秩下保持稳定。 相比之下，Metis 测试集从 $k=16$ 到 $k=64$ 仍在持续提升。 这种差异表明，两个记忆操作基准需要不同水平的记忆容量。

Across all datasets, performance becomes close to the full model at $k=64$. The overall recovery reaches 99.9%, while further increasing the rank brings little additional benefit. These results confirm that the memory states still contain substantial redundancy and that most useful information lies in a relatively low-dimensional subspace. They also show that the appropriate compression level depends on the task, since different benchmarks require different amounts and types of memory information.

在所有数据集上，当 $k=64$ 时，性能都接近完整模型。 整体恢复比例达到 99.9%，进一步增加秩几乎不会带来额外收益。 这些结果证实，记忆状态中仍包含大量冗余，大多数有用信息位于相对低维的子空间中。 它们还表明，合适的压缩程度取决于任务，因为不同基准需要不同数量和类型的记忆信息。

6.9 Case Studies

图5：Metis 在不同对话场景中的案例研究。

We conduct qualitative case studies to show the behavior of Metis-4B under different scenarios. At each turn, the model first generates a response based on the current input and existing memory, and then updates the user input in the memory state. We present several representative cases in Figure 5 to qualitatively examine the native memory behaviors of Metis-4B. Each case starts from an empty native memory state, and the model must use information stored in previous interaction steps.

我们开展定性案例研究，以展示 Metis-4B 在不同场景下的行为。 在每一轮中，模型首先根据当前输入和已有记忆生成响应，然后把用户输入更新进记忆状态。 我们在图5中给出若干代表性案例，以定性考察 Metis-4B 的原生记忆行为。 每个案例都从空的原生记忆状态开始，模型必须使用先前交互步骤中存储的信息。

In the remembering case, Metis correctly stores Alice's food preference and retrieves it in a later query. In the multi-fact case, the model retains several attributes about Alice and correctly selects her age after other attributes are introduced. This result suggests that Metis can bind different values to their corresponding attributes and reduce interference among related facts. The distractor case further shows that an unrelated dialogue turn does not overwrite the stored preference. Metis can therefore distinguish useful memory from ordinary conversational content.

在记住案例中，Metis 正确存储 Alice 的食物偏好，并在后续查询中将其取回。 在多事实案例中，模型保留 Alice 的多个属性，并在引入其他属性后正确选择她的年龄。 这一结果表明，Metis 可以把不同的值绑定到其对应属性，并减少相关事实之间的干扰。 干扰项案例进一步表明，无关的对话轮次不会覆写已存储的偏好。 因此，Metis 能够区分有用记忆与普通对话内容。

The forgetting case demonstrates that the native memory state is not append-only. After receiving a forgetting instruction, Metis no longer provides the removed preference in the subsequent query. This indicates that the model can modify its latent memory state according to the semantic intent of an instruction. However, the immediate response to the forgetting instruction still repeats the old fact instead of explicitly confirming its removal. The final memory state is correct, but the response at the operation step is not fully aligned with the intended memory operation.

遗忘案例表明，原生记忆状态并非只能追加。 接收到遗忘指令后，Metis 在后续查询中不再给出已移除的偏好。 这说明模型可以根据指令的语义意图修改其潜在记忆状态。 然而，对遗忘指令的即时响应仍然重复了旧事实，而不是明确确认已将其移除。 最终记忆状态是正确的，但操作步骤中的响应并未与预期记忆操作完全对齐。

This behavior may result from the current step over-emphasizing previous memory states. Overall, these cases show the effectiveness of Metis, while also revealing room for improvement in its consistency.

这种行为可能源于当前步骤过度强调先前记忆状态。 总体而言，这些案例展示了 Metis 的有效性，同时也表明其一致性仍有改进空间。

7. Related Work

7.1 Memory of LLMs and Agents

In recent years, large foundation models and agents have been widely applied to fields such as personal assistants, deep research, and coding agents. A critical capability of these systems is memory, which stores past information to support future inference. Based on their representation forms, memory mechanisms of large foundation models and agents are generally categorized into three types, including textual memory, latent memory, and parametric memory. Textual memory typically represents information as text, relying on RAG for storage and retrieval. These methods provide information for backbones to support inference by In-Context Learning (ICL).

近年来，大型基础模型和智能体已广泛应用于个人助理、深度研究和编程智能体等领域。 这些系统的一项关键能力是记忆，它存储过去的信息以支持未来推理。 根据表示形式，大型基础模型和智能体的记忆机制通常分为文本记忆、潜在记忆和参数化记忆三类。 文本记忆通常以文本形式表示信息，依靠 RAG 进行存储和检索。 这些方法为骨干提供信息，通过上下文学习支持推理。

For example, MemoryBank proposes a hierarchical storage approach with dual-tower dense retrieval to maintain historical conversations with users. MemTree designs a tree-structured memory mechanism to model the abstraction levels of information, which dynamically updates based on semantic embeddings. In contrast, latent memory captures memory through intermediate activations of models. For example, NextMem compresses factual memory into latent representations through an autoregressive autoencoder, while MemGen generates latent memory tokens that are interwoven into the reasoning process.

例如，MemoryBank 提出一种结合双塔稠密检索的分层存储方法，用于维护与用户的历史对话。 MemTree 设计树结构记忆机制来建模信息的抽象层级，并根据语义嵌入动态更新。 相比之下，潜在记忆通过模型的中间激活捕获记忆。 例如，NextMem 通过自回归自动编码器把事实记忆压缩为潜在表示，而 MemGen 生成交织到推理过程中的潜在记忆 token。

Additionally, parametric memory injects knowledge into internal model parameters. For example, Locas views the FFN as a soft look-up table. By adding a bypass FFN, it stores test-time information from a key-value perspective. Furthermore, knowledge editing can also be considered a parametric memory method. ROME treats the projection matrix as an associative memory and inserts a new factual association through a rank-one update. Although textual memory remains the most effective approach in industry, latent memory and parametric memory are emerging as promising research directions.

此外，参数化记忆把知识注入模型内部参数。 例如，Locas 把 FFN 视为软查找表。 它通过增加旁路 FFN，从键值视角存储测试时信息。 此外，知识编辑也可以视为一种参数化记忆方法。 ROME 把投影矩阵视为联想记忆，并通过秩一更新插入新的事实关联。 尽管文本记忆仍是工业界最有效的方法，潜在记忆和参数化记忆正成为很有前景的研究方向。

7.2 Fast Weight Programming

Recently, FWP has attracted widespread attention. This paradigm not only uses parameters learned during training (i.e., slow weights), but also maintains dynamic parameters (i.e., fast weights) during inference to capture sequence-dependent information. Existing methods in this line of work generally follow several main directions. Linear attention replaces the softmax kernel with feature maps to achieve linear complexity and a recurrent state. In addition, it has been shown that linear transformers are secretly fast weight programmers.

近来，快速权重编程（FWP）受到广泛关注。 该范式不仅使用训练期间学到的参数（即慢权重），还在推理期间维护动态参数（即快权重），以捕获依赖序列的信息。 这一方向的现有方法通常遵循若干主要路线。 线性注意力使用特征映射替换 softmax 核，以实现线性复杂度和循环状态。 此外，已有研究表明，线性 Transformer 本质上是快速权重编程器。

Subsequent works enrich the update rule, such as RetNet and RWKV. Furthermore, state space models compress a sequence into a fixed-size recurrent state with linear-time computation, such as S4 and Mamba, while Mamba-2 further reveals a duality between state space models and attention. TTT also treats the recurrent state as fast weights that are optimized by self-supervised gradient descent during inference, such as the TTT layer and Titans.

后续工作进一步丰富了更新规则，例如 RetNet 和 RWKV。 此外，S4 和 Mamba 等状态空间模型以线性时间计算把序列压缩为固定大小的循环状态，而 Mamba-2 进一步揭示了状态空间模型与注意力之间的对偶性。 TTT 也把循环状态视为快权重，在推理期间通过自监督梯度下降进行优化，例如 TTT layer 和 Titans。

7.3 Memory-Augmented Neural Networks

MANNs introduce explicit memory modules to improve a model's ability to store and retrieve task-specific information during inference. Early work, such as Memory Networks and Neural Turing Machines, augments neural controllers with external memory and learns differentiable read and write operations over memory slots. These methods show that neural models can use non-parametric memory to support associative recall, algorithmic reasoning, and few-shot adaptation. However, their memory is usually maintained as a separate storage module, and the memory procedures are often designed independently from the backbone computation.

记忆增强神经网络（MANN）引入显式记忆模块，以提升模型在推理期间存储和检索任务特定信息的能力。 Memory Networks 和 Neural Turing Machines 等早期工作使用外部记忆增强神经控制器，并学习对记忆槽执行可微的读写操作。 这些方法表明，神经模型可以使用非参数记忆支持联想回忆、算法推理和少样本适应。 然而，它们的记忆通常作为独立存储模块维护，记忆过程也往往独立于骨干计算进行设计。

Recent models also maintain dynamic states during inference, such as recurrent memory. Unlike static model parameters learned during training, these dynamic states are updated according to the current input sequence and capture information that changes over time. Our work follows this general direction, but focuses on integrating memory storage and utilization directly into the model computation, so that the model can maintain sequence-dependent information more natively.

近期模型也会在推理期间维护动态状态，例如循环记忆。 与训练期间学到的静态模型参数不同，这些动态状态根据当前输入序列更新，并捕获随时间变化的信息。 我们的工作遵循这一总体方向，但重点把记忆存储与利用直接集成到模型计算中，使模型能够以更原生的方式维护依赖序列的信息。

8. Conclusion

In this paper, we introduce memory foundation models and provide formal definitions of native memory based on the memory state and memory procedures. Based on this formulation, we propose Metis, the first prototype of memory foundation models. We introduce Metis blocks composed of local memory blocks and hyper memory blocks, enabling the model to maintain compact dense memory states across interaction steps and to update them according to the current input and generated response. We further construct a memory-specific dataset from public benchmarks and design a mid-training framework with memory reconstruction, memory operation, and regularization objectives. Our experiments verify the effectiveness of Metis and analyze its behavior from multiple perspectives.

在本文中，我们引入记忆基础模型，并基于记忆状态和记忆过程给出原生记忆的形式化定义。 基于这一形式化框架，我们提出 Metis，这是记忆基础模型的首个原型。 我们引入由局部记忆块和超记忆块组成的 Metis 块，使模型能够跨交互步骤维护紧凑的稠密记忆状态，并根据当前输入和生成响应对其进行更新。 我们还从公开基准构建记忆专项数据集，并设计包含记忆重构、记忆操作和正则化目标的中期训练框架。 我们的实验验证了 Metis 的有效性，并从多个角度分析了其行为。

Despite these promising results, Metis is still an early step toward memory foundation models. Since the current native memory state compresses information into fixed-size latent parameters, performance may degrade in extremely long-term scenarios, and semantically similar facts may sometimes be confused in the latent space. Therefore, native memory still cannot be viewed as a complete replacement for external memory. Instead, we believe it opens a complementary direction for building future foundation models with more efficient, optimizable, and deeply integrated memory capabilities. Future work may further improve memory capacity, controllability, and interpretability, explore hybrid systems that combine native and external memory, and scale native memory training to broader domains and longer interactions.

尽管这些结果很有前景，Metis 仍只是迈向记忆基础模型的早期一步。 由于当前原生记忆状态把信息压缩进固定大小的潜在参数，在极长期场景中性能可能下降，语义相似的事实有时也可能在潜在空间中混淆。 因此，原生记忆仍不能被视为外部记忆的完整替代品。 相反，我们认为它开辟了一条互补方向，可用于构建具有更高效、可优化且深度集成的记忆能力的未来基础模型。 未来工作可以进一步提高记忆容量、可控性和可解释性，探索结合原生记忆与外部记忆的混合系统，并把原生记忆训练扩展到更广泛的领域和更长的交互。