---

MemT-1.0: Towards Memory-Native Evolution

MemoryAgent

MemTensor Team. MemT-1.0: Towards Memory-Native Evolution. 2026.

https://github.com/MemTensor/MemT-1.0

https://huggingface.co/collections/MemTensor/memt10-1

---

MemT-1.0：迈向记忆原生演化

Abstract

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity.

记忆原生演化要求将记忆变为基础模型的内在能力，使模型能够持续积累和利用信息。 为此，我们提出 MemT-1.0，这是一种将记忆内化为模型计算组成部分的新型记忆基础模型架构。 在 Metis 的基础上，MemT-1.0 采用 Gated DeltaNet-2，在每轮交互内使用稳定的 token 级记忆更新机制。 我们进一步引入混合存储（Mixture-of-Storage），以扩展原生记忆容量。

Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately $1.86\times$ end-to-end inference speedup and a $66\times$ reduction in persistent storage relative to full context KV cache.

经过精心的骨干模型准备后，我们构建了由持续预训练和后训练组成的两阶段训练流程，并以 34B token 的训练数据及高效的训练、推理基础设施提供支持。 我们发布了包含 MemT-1.0-Nano 与 MemT-1.0-Flash 的模型家族，并全面评估其记忆能力和通用能力。 结果表明，它们的平均表现优于参数化记忆基线，在与检索增强方法比较时也具有竞争力，同时基本保留了通用能力。 当历史长度为 128K token 时，相较完整上下文 KV 缓存，MemT-1.0-Nano 的端到端推理速度约提升 $1.86\times$，持久存储需求减少至原来的 $1/66$。

By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.

通过内化检索得到的程序性经验，MemT-1.0 提高了智能体任务的成功率，为记忆原生演化迈出了切实的一步。

图1：MemT-1.0 在代表性基准上的记忆性能。

图2：经过骨干模型准备、未使用原生记忆的 MemT-1.0-Flash 的智能体性能。

图3：MemT-1.0-Nano 的推理效率与存储开销。

1. Introduction

The rapid evolution of foundation models is expanding the boundary of what can be natively learned and performed within a single model. Many capabilities are progressively internalized from external systems into the model itself, such as reasoning and multimodality. Memory represents another fundamental capability, enabling models to accumulate experience and adapt over extended interactions. Our previous work, Metis, first introduced memory foundation models to internalize memory as a native capability of foundation models. This offers a path beyond external context and enables models to continuously accumulate information and experience, which marks a critical step towards memory-native evolution.

基础模型的快速演进正在拓展单个模型能够原生学习和执行的能力边界。 许多能力正逐步从外部系统内化到模型本身，例如推理能力和多模态能力。 记忆是另一项基础能力，使模型能够在长期交互中积累经验并进行适应。 我们此前的工作 Metis 首次提出记忆基础模型，将记忆内化为基础模型的原生能力。 这提供了一条超越外部上下文的路径，使模型能够持续积累信息与经验，标志着迈向记忆原生演化的关键一步。

To improve memory capabilities, we introduce MemT-1.0, a new architecture with substantial upgrades over Metis, while retaining the structure of dynamic local memory and trainable hyper memory. It replaces token selection with a stable token-level update mechanism based on Gated DeltaNet-2 (GDN-2). To further scale native memory capability, we introduce Mixture-of-Storage (MoS) to expand the capacity of native memory. We release a model family comprising MemT-1.0-Nano with 36B parameters (3B activated, 10M dynamic memory) and MemT-1.0-Flash with 288B parameters (17B activated, 336M dynamic memory).

为提升记忆能力，我们提出 MemT-1.0：这一新架构在保留动态局部记忆与可训练超记忆结构的同时，对 Metis 进行了实质性升级。 它使用基于 Gated DeltaNet-2（GDN-2）的稳定 token 级更新机制，替代 token 选择。 为进一步扩展原生记忆能力，我们引入混合存储（MoS），以增大原生记忆容量。 我们发布了一个模型家族：MemT-1.0-Nano 具有 36B 参数（激活 3B，动态记忆 10M），MemT-1.0-Flash 具有 288B 参数（激活 17B，动态记忆 336M）。

Following a deliberate backbone preparation, we develop a two-stage training pipeline for MemT-1.0. The first stage performs continued pre-training (CPT) to establish language modeling capabilities with native memory. The post-training stage further adapts the model to downstream memory tasks while preserving its general capabilities. To support this training pipeline, we construct a large-scale dataset of 34B tokens.

经过精心的骨干模型准备后，我们为 MemT-1.0 开发了两阶段训练流程。 第一阶段进行持续预训练（CPT），建立带有原生记忆的语言建模能力。 后训练阶段进一步使模型适应下游记忆任务，同时保留其通用能力。 为支持这一训练流程，我们构建了一个包含 34B token 的大规模数据集。

Across 11 memory benchmarks, our models outperform parametric memory baselines on average, showing effective utilization of historical information without accessing the original context during inference. They also largely preserve general capabilities under the noise setting. Our evaluation shows an increasing end-to-end inference latency and storage overhead advantage over full context on longer histories.

在 11 个记忆基准上，我们的模型平均优于参数化记忆基线，表明它们能够在推理时不访问原始上下文的情况下，有效利用历史信息。 在噪声设置下，它们也基本保留了通用能力。 评估显示，随着历史变长，相较完整上下文，它们在端到端推理延迟和存储开销方面的优势不断扩大。

To support training and deployment, we develop a distributed infrastructure, which preserves memory isolation across independent trajectories. We organize stateful inference around separate memory storage and utilization procedures, with shared model parameters and trajectory-specific memory states. For a fixed memory configuration, the persistent state size remains constant as interactions accumulate. In addition, we explore the scaling behavior of MemT-1.0. Unlike general foundation models, memory foundation models introduce memory capacity as a critical scaling dimension, so we conduct scaling studies over memory capacity and training data. These findings provide valuable guidance for building stronger memory foundation models.

为支持训练与部署，我们开发了分布式基础设施，使相互独立的轨迹之间保持记忆隔离。 我们围绕相互分离的记忆存储和记忆利用过程来组织有状态推理，模型参数共享，而记忆状态归各条轨迹所有。 在记忆配置固定时，持久状态的大小不会随交互累积而增长。 此外，我们还探索了 MemT-1.0 的扩展行为。 与通用基础模型不同，记忆基础模型将记忆容量引入为一个关键扩展维度，因此我们围绕记忆容量与训练数据开展了扩展研究。 这些发现为构建更强的记忆基础模型提供了有价值的指导。

We further apply MemT-1.0 to experience-guided coding agents, where retrieved procedural experience is internalized into native memory to guide subsequent actions. On SWE-bench Verified and SWE-bench Multilingual, our method improves average task success and consistency across repeated runs over agents without experience, while achieving performance comparable to explicit textual experience. These findings extend the role of native memory from information recall to procedural guidance in multi-step execution.

我们进一步将 MemT-1.0 应用于经验引导的编程智能体，把检索得到的程序性经验内化到原生记忆中，以指导后续动作。 在 SWE-bench Verified 和 SWE-bench Multilingual 上，相较不使用经验的智能体，我们的方法提高了平均任务成功率及重复运行的一致性，同时取得了与显式文本经验相当的性能。 这些发现将原生记忆的作用从信息回忆，拓展到了多步执行中的程序性指导。

Our main contributions are summarized as follows:

我们的主要贡献概括如下：

• We introduce MemT-1.0, a memory foundation model architecture with improved memory storage and utilization mechanisms, together with MoS for scalable memory capacity.
• Following a deliberate backbone preparation, we design a two-stage training pipeline including CPT and post-training stages, with a corresponding infrastructure design. We construct a training corpus of 34B tokens, and systematically evaluate MemT-1.0 on both memory and general benchmarks. The results show that MemT-1.0 achieves superior memory performance while largely preserving general capabilities.
• We explore the scaling behavior of memory capabilities in MemT-1.0. We further deploy MemT-1.0 in practical agentic coding tasks to demonstrate its effectiveness and potential in enabling memory-native evolution. We also discuss its current limitations and promising directions for future development.

• 我们提出 MemT-1.0，这是一种改进了记忆存储与利用机制的记忆基础模型架构，并提供 MoS 以实现可扩展的记忆容量。
• 经过精心的骨干模型准备后，我们设计了包含 CPT 和后训练阶段的两阶段训练流程，并配套设计了基础设施。我们构建了 34B token 的训练语料，并在记忆基准和通用基准上系统评估 MemT-1.0。结果表明，MemT-1.0 在基本保留通用能力的同时，取得了更优的记忆表现。
• 我们探索了 MemT-1.0 记忆能力的扩展行为。我们进一步将 MemT-1.0 部署到实际的智能体编程任务中，展示其有效性，以及推动记忆原生演化的潜力。我们还讨论了当前局限和未来值得探索的发展方向。

2. Model Architecture

图4：MemT-1.0 的架构。MemT-1.0 为因果语言模型赋予原生记忆，使其随交互步骤演化。每个选定的 Transformer 块都增加一个原生记忆块，并行执行原始注意力与记忆注意力。记忆机制由动态局部记忆和可训练超记忆组成，并通过原生记忆存储与利用过程更新、检索持久信息。

2.1. Overview

MemT-1.0 is formulated under a multi-step scenario with discrete steps $t\in \{1,2,\cdots ,T\}$, as presented in Figure 4. At step $t$, let ${\theta }_t$ denote the current model parameters. Given the input $X_t$, the autoregressive generation process can be formulated as

MemT-1.0 在具有离散步骤 $t\in \{1,2,\cdots ,T\}$ 的多步场景下进行定义，如 图4 所示。 在步骤 $t$，用 ${\theta }_t$ 表示当前模型参数。 给定输入 $X_t$，自回归生成过程可表示为

$$y_{t,k}\sim P(\cdot |X_t,Y_{t,<k};{\theta }_t),$$

where $Y_{t,<k}$ denotes the tokens generated before $y_{t,k}$. Meanwhile, the stateful model parameter is updated by

其中，$Y_{t,<k}$ 表示在 $y_{t,k}$ 之前生成的 token。 与此同时，有状态模型的参数通过下式更新：

$${\theta }_{t+1}\sim P(\cdot |X_t,Y_t;{\theta }_t),$$

where $Y_t$ is all generated tokens at step $t$. MemT-1.0 is built upon general causal language models. We denote the input sequence as $S_{t,k}=X_t||Y_{t,<k}=(x_{t,1},\cdots ,y_{t,1},\cdots ,y_{t,k-1})$ with length $L_{t,k}$. First, the input is mapped to the initial hidden state ${\mathbf{H}}_{t,k}^{(0)}\in {\mathbb{R}}^{L_{t,k}\times d}$ through the embedding layer, where $d$ is the dimension of the hidden state. Then, it is processed sequentially by $L$ stacked Transformer blocks as follows:

其中，$Y_t$ 为步骤 $t$ 生成的全部 token。 MemT-1.0 构建在通用因果语言模型之上。 我们将输入序列记为 $S_{t,k}=X_t||Y_{t,<k}=(x_{t,1},\cdots ,y_{t,1},\cdots ,y_{t,k-1})$，其长度为 $L_{t,k}$。 首先，通过嵌入层将输入映射为初始隐藏状态 ${\mathbf{H}}_{t,k}^{(0)}\in {\mathbb{R}}^{L_{t,k}\times d}$，其中 $d$ 是隐藏状态的维度。 随后，输入依次经过 $L$ 个堆叠的 Transformer 块处理，具体如下：

$$\begin{array}{c}{\mathbf{H}}_{t,k}^{(l)}=\mathrm{TransformerBlock}\left({\mathbf{H}}_{t,k}^{(l-1)}\right),\\ \text{for }l=1,2,\cdots L.\end{array}$$

After that, the last hidden state ${\mathbf{H}}_{t,k}^{(L)}$ is normalized and projected into the vocabulary space through the language modeling head ${\mathbf{W}}_{\text{LM}}\in {\mathbb{R}}^{d\times |\mathcal{V}|}$, and a softmax function is then applied to obtain the probability distribution by

之后，对最后的隐藏状态 ${\mathbf{H}}_{t,k}^{(L)}$ 进行归一化，并通过语言建模头 ${\mathbf{W}}_{\text{LM}}\in {\mathbb{R}}^{d\times |\mathcal{V}|}$ 投影到词表空间，再应用 softmax 函数得到概率分布：

$$\begin{array}{c}P(\cdot |X_t,Y_{t,<k};{\theta }_t)=\\ \mathrm{Softmax}\left(\mathrm{Norm}\left({\mathbf{h}}_{t,k}^{(L)}\right){\mathbf{W}}_{\text{LM}}\right),\end{array}$$

where $|\mathcal{V}|$ is the vocabulary size and ${\mathbf{h}}_{t,k}^{(L)}$ is the last position of ${\mathbf{H}}_{t,k}^{(L)}$.

其中，$|\mathcal{V}|$ 为词表大小，${\mathbf{h}}_{t,k}^{(L)}$ 为 ${\mathbf{H}}_{t,k}^{(L)}$ 的最后一个位置。

MemT-1.0 augments each selected standard Transformer block with a native memory block, which is responsible for maintaining the memory from previous information. For the $l$-th Transformer block, we denote the computation of attention output as

MemT-1.0 为每个选定的标准 Transformer 块增加一个原生记忆块，负责维护来自以往信息的记忆。 对于第 $l$ 个 Transformer 块，我们将注意力输出的计算表示为

$$\begin{array}{c}{\mathbf{A}}_{\text{ori},t,k}^{(l)}=\mathrm{Attention}\left({\tilde{\mathbf{H}}}_{t,k}^{(l)}\right),\\ {\tilde{\mathbf{H}}}_{t,k}^{(l)}=\mathrm{Pre-Attention}\left({\mathbf{H}}_{t,k}^{(l-1)}\right),\end{array}$$

where $\mathrm{Attention}(\cdot )$ represents the backbone attention and ${\tilde{\mathbf{H}}}^{(l)}$ indicates the hidden state after pre-attention operations, such as pre-normalization. MemT-1.0 introduces a parallel memory attention mechanism to utilize the parametric memory accumulated from previous steps for targeted layers, by

其中，$\mathrm{Attention}(\cdot )$ 表示骨干注意力，${\tilde{\mathbf{H}}}^{(l)}$ 表示经过预归一化等注意力前置操作后的隐藏状态。 MemT-1.0 引入并行记忆注意力机制，在目标层利用前面步骤积累的参数化记忆，具体为

$$\begin{array}{c}{\mathbf{A}}_{t,k}^{(l)}={\mathbf{A}}_{\text{ori},t,k}^{(l)}+{\mathbf{A}}_{\text{mem},t,k}^{(l)},\\ {\mathbf{A}}_{\text{mem},t,k}^{(l)}=\mathrm{MemoryAttention}({\tilde{\mathbf{H}}}_{t,k}^{(l)};\cdot ),\end{array}$$

where ${\mathbf{A}}_{\text{mem},t,k}^{(l)}$ indicates the output of memory attention. After that, the combined attention output ${\mathbf{A}}_{t,k}^{(l)}$ is used in the subsequent computation as

其中，${\mathbf{A}}_{\text{mem},t,k}^{(l)}$ 表示记忆注意力的输出。 随后，将合并后的注意力输出 ${\mathbf{A}}_{t,k}^{(l)}$ 用于后续计算：

$${\mathbf{H}}_{t,k}^{(l)}=\mathrm{Post-Attention}({\mathbf{A}}_{t,k}^{(l)},{\mathbf{H}}_{t,k}^{(l-1)}).$$

For different backbones, we preserve the original $\mathrm{Pre-Attention}(\cdot )$, $\mathrm{Attention}(\cdot )$, and $\mathrm{Post-Attention}(\cdot )$ unchanged to retain the backbone capabilities, and introduce memory through $\mathrm{MemoryAttention}(\cdot )$ for different backbones. Specifically, we use Qwen3.6-35B-A3B, as released, as the backbone of MemT-1.0-Nano and a further post-trained DeepSeek-V4-Flash (Section 3.1) as the backbone of MemT-1.0-Flash.

对于不同骨干模型，我们保留原有的 $\mathrm{Pre-Attention}(\cdot )$、$\mathrm{Attention}(\cdot )$ 和 $\mathrm{Post-Attention}(\cdot )$，以维持骨干模型的能力，并通过 $\mathrm{MemoryAttention}(\cdot )$ 为不同骨干引入记忆。 具体来说，我们直接使用发布的 Qwen3.6-35B-A3B 作为 MemT-1.0-Nano 的骨干，并使用进一步后训练的 DeepSeek-V4-Flash（第 3.1 节）作为 MemT-1.0-Flash 的骨干。

2.2. Native Memory Block

2.2.1. Native Memory State

Following our previous work Metis, a native memory block in MemT-1.0 comprises a local memory block and a hyper memory block. The former stores online information, while the latter is trained to equip general foundation models with memory capabilities in a data-driven manner.

沿用我们此前的工作 Metis，MemT-1.0 的原生记忆块由局部记忆块与超记忆块组成。 前者存储在线信息，后者通过训练，以数据驱动的方式为通用基础模型赋予记忆能力。

Local Memory Block. In MemT-1.0, the dynamic memory state at the $t$-th step and $l$-th layer can be represented as ${\mathbf{M}}_t^{(l)}\in {\mathbb{R}}^{H_{kv}\times d_k\times d_v}$, where $d_k,d_v$ represent the dimensions of memory keys and values, and $H_{kv}$ denotes the number of memory channels. Compared with Metis, we remove the query-key normalization vector and extend the memory state to multiple channels. We initialize ${\mathbf{M}}_t^{(l)}=\mathbf{0}$ at the start of new trajectories.

局部记忆块。 在 MemT-1.0 中，第 $t$ 步、第 $l$ 层的动态记忆状态可表示为 ${\mathbf{M}}_t^{(l)}\in {\mathbb{R}}^{H_{kv}\times d_k\times d_v}$，其中 $d_k,d_v$ 表示记忆键和值的维度，$H_{kv}$ 表示记忆通道数。 与 Metis 相比，我们移除了查询—键归一化向量，并将记忆状态扩展到多个通道。 在新轨迹开始时，我们将记忆初始化为 ${\mathbf{M}}_t^{(l)}=\mathbf{0}$。

Hyper Memory Block. In the hyper memory block, the multi-channel projections of memory query, key, and value are denoted as ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{H_q\times d\times d_k}$, ${\tilde{\mathbf{W}}}_K^{(l)}\in {\mathbb{R}}^{H_{kv}\times d\times d_k}$ and ${\tilde{\mathbf{W}}}_V^{(l)}\in {\mathbb{R}}^{H_{kv}\times d\times d_v}$ for each layer $l$. Beyond these core memory projections, this block includes all other trainable parameters for memory processing. First, the gating projections include the erase gate ${\mathbf{W}}_{\beta }^{(l)}\in {\mathbb{R}}^{d\times H_{kv}\times d_k}$ and the write gate ${\mathbf{W}}_{\omega }^{(l)}\in {\mathbb{R}}^{d\times H_{kv}\times d_v}$, along with the forget gate components consisting of ${\mathbf{W}}_f^{(l)}\in {\mathbb{R}}^{d\times d_{\text{gate}}}$, ${\mathbf{W}}_{\lambda }^{(l)}\in {\mathbb{R}}^{d_{\text{gate}}\times H_{kv}\times d_k}$, ${\mathbf{b}}_{\lambda }^{(l)}\in {\mathbb{R}}^{H_{kv}\times d_k}$, and a learnable vector ${\mathbf{s}}^{(l)}\in {\mathbb{R}}^{H_{kv}}$. Finally, the memory output gating is parameterized by a shared down-projection ${\mathbf{W}}_{g/d}^{(l)}\in {\mathbb{R}}^{d\times d_{g/d}}$ and a multi-channel up-projection ${\mathbf{W}}_{g/u}^{(l)}\in {\mathbb{R}}^{H_q\times d_{g/d}\times d_v}$. Generally, we have $H_q=H_{kv}$ to align query channel and key-value channels. These trainable parameters are denoted as $\varphi$, which can be optimized during the training phase and remain fixed during online interaction.

超记忆块。 在超记忆块中，对每一层 $l$，记忆查询、键和值的多通道投影分别记为 ${\tilde{\mathbf{W}}}_Q^{(l)}\in {\mathbb{R}}^{H_q\times d\times d_k}$、${\tilde{\mathbf{W}}}_K^{(l)}\in {\mathbb{R}}^{H_{kv}\times d\times d_k}$ 和 ${\tilde{\mathbf{W}}}_V^{(l)}\in {\mathbb{R}}^{H_{kv}\times d\times d_v}$。 除了这些核心记忆投影外，该块还包含记忆处理所需的全部其他可训练参数。 首先，门控投影包括擦除门 ${\mathbf{W}}_{\beta }^{(l)}\in {\mathbb{R}}^{d\times H_{kv}\times d_k}$ 和写入门 ${\mathbf{W}}_{\omega }^{(l)}\in {\mathbb{R}}^{d\times H_{kv}\times d_v}$，以及遗忘门组件 ${\mathbf{W}}_f^{(l)}\in {\mathbb{R}}^{d\times d_{\text{gate}}}$、${\mathbf{W}}_{\lambda }^{(l)}\in {\mathbb{R}}^{d_{\text{gate}}\times H_{kv}\times d_k}$、${\mathbf{b}}_{\lambda }^{(l)}\in {\mathbb{R}}^{H_{kv}\times d_k}$ 和可学习向量 ${\mathbf{s}}^{(l)}\in {\mathbb{R}}^{H_{kv}}$。 最后，记忆输出门控由共享的降维投影 ${\mathbf{W}}_{g/d}^{(l)}\in {\mathbb{R}}^{d\times d_{g/d}}$ 和多通道升维投影 ${\mathbf{W}}_{g/u}^{(l)}\in {\mathbb{R}}^{H_q\times d_{g/d}\times d_v}$ 参数化。 通常设 $H_q=H_{kv}$，使查询通道与键值通道对齐。 这些可训练参数记为 $\varphi$，在训练阶段可以优化，在线交互期间则保持固定。

In summary, the model parameters ${\theta }_t$ at step $t$ include the backbone parameters ${\theta }_b$, the static but trainable parameters $\varphi$, and the dynamic memory states $\{{\mathbf{M}}_t^{(l)}{\}}_{l=1}^L$. During online inference, these dynamic memory states are responsible for storing online information and evolve over interactions.

概括而言，步骤 $t$ 的模型参数 ${\theta }_t$ 包括骨干参数 ${\theta }_b$、静态但可训练的参数 $\varphi$，以及动态记忆状态 $\{{\mathbf{M}}_t^{(l)}{\}}_{l=1}^L$。 在线推理时，这些动态记忆状态负责存储在线信息，并随着交互演化。

2.2.2. Native Memory Procedures

MemT-1.0 consists of two parallel native procedures, including memory storage and memory utilization. The memory storage procedure updates the dynamic memory states to preserve information from the current interaction, while the memory utilization procedure retrieves information from previously accumulated memory states to support autoregressive generation.

MemT-1.0 包含两个并行的原生过程：记忆存储与记忆利用。 记忆存储过程更新动态记忆状态，以保存当前交互的信息；记忆利用过程则从此前积累的记忆状态中检索信息，支持自回归生成。

Native Memory Storage Procedure. MemT-1.0 replaces the token selection with a stable token-level update mechanism based on GDN-2 to update the dynamic memory state in each turn. As the memory storage procedure is triggered at the turn level, we denote the hidden states as ${\mathbf{H}}_t^{(l)}$ of $S_t=X_t||Y_t$. For the pre-attention input ${\tilde{\mathbf{H}}}_t^{(l)}$, we compute the memory key and value of the $h$-th channel by

原生记忆存储过程。 MemT-1.0 使用基于 GDN-2 的稳定 token 级更新机制，替代 token 选择，以在每轮交互中更新动态记忆状态。 由于记忆存储过程在轮次级别触发，我们将 $S_t=X_t||Y_t$ 的隐藏状态记为 ${\mathbf{H}}_t^{(l)}$。 对于注意力前置输入 ${\tilde{\mathbf{H}}}_t^{(l)}$，第 $h$ 个通道的记忆键和值计算如下：

$$\begin{array}{c}{\mathbf{K}}_{t,h}^{(l)}=\mathrm{L}2\mathrm{Norm}\left({\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_{K,h}^{(l)}\right),\\ {\mathbf{V}}_{t,h}^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_{V,h}^{(l)},\end{array}$$

where ${\tilde{\mathbf{W}}}_{K,h}^{(l)}$ and ${\tilde{\mathbf{W}}}_{V,h}^{(l)}$ are the key and value projection matrices for the $h$-th memory channel. Then, the erase gate and write gate can be obtained by

其中，${\tilde{\mathbf{W}}}_{K,h}^{(l)}$ 和 ${\tilde{\mathbf{W}}}_{V,h}^{(l)}$ 是第 $h$ 个记忆通道的键投影矩阵和值投影矩阵。 接着，擦除门与写入门可通过下式得到：

$$\begin{array}{c}{\mathit{\beta }}_t^{(l)}=\sigma \left({\tilde{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_{\beta }^{(l)}\right),\\ {\mathit{\omega }}_t^{(l)}=\sigma \left({\tilde{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_{\omega }^{(l)}\right),\end{array}$$

where ${\mathbf{W}}_{\beta }^{(l)},{\mathbf{W}}_{\omega }^{(l)}$ are gating projection matrices. In addition, the forget gate is calculated by

其中，${\mathbf{W}}_{\beta }^{(l)},{\mathbf{W}}_{\omega }^{(l)}$ 为门控投影矩阵。 此外，遗忘门的计算为

$$\begin{array}{c}{\mathit{\lambda }}_{t,h}^{(l)}=\\ -\exp \left({\mathbf{s}}_h^{(l)}\right)\odot \mathrm{Softplus}({\mathbf{F}}_t^{(l)}{\mathbf{W}}_{\lambda ,h}^{(l)}+{\mathbf{b}}_{\lambda ,h}^{(l)}),\\ {\mathbf{F}}_t^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_f^{(l)},\end{array}$$

where ${\mathbf{W}}_f^{(l)},{\mathbf{W}}_{\lambda }^{(l)},{\mathbf{b}}_{\lambda }^{(l)},{\mathbf{s}}_h^{(l)}$ are forget-gate parameters. To prevent the divergence of memory states during updates, we introduce a stable mechanism for the token-level update, where it first calculates

其中，${\mathbf{W}}_f^{(l)},{\mathbf{W}}_{\lambda }^{(l)},{\mathbf{b}}_{\lambda }^{(l)},{\mathbf{s}}_h^{(l)}$ 为遗忘门参数。 为防止记忆状态在更新过程中发散，我们为 token 级更新引入了稳定机制，首先计算

$${\mathbf{U}}_{t,i}^{(l)}=\sqrt{{\mathit{\beta }}_{t,i}^{(l)}}\odot {\mathbf{K}}_{t,i}^{(l)}.$$

Then, the memory state can be updated by

随后，记忆状态可更新为

$$\begin{array}{c}{\tilde{\mathbf{M}}}_{t,i,h}^{(l)}=\\ [\mathbf{I}-{\mathbf{U}}_{t,i,h}^{(l)\mathrm{\top }}{\mathbf{U}}_{t,i,h}^{(l)}]\cdot {\mathbf{D}}_{t,i,h}^{(l)}{\tilde{\mathbf{M}}}_{t,i-1,h}^{(l)}\\ +{\mathbf{U}}_{t,i,h}^{(l)\mathrm{\top }}({\mathit{\omega }}_{t,i,h}^{(l)}\odot {\mathbf{V}}_{t,i,h}^{(l)}),\end{array}$$

where the initialization is given by ${\tilde{\mathbf{M}}}_{t,0,h}^{(l)}={\mathbf{M}}_{t,h}^{(l)}$, and the state transition is defined as ${\mathbf{M}}_{t+1,h}^{(l)}={\tilde{\mathbf{M}}}_{t,L_t,h}^{(l)}$, with

其中，初始状态设为 ${\tilde{\mathbf{M}}}_{t,0,h}^{(l)}={\mathbf{M}}_{t,h}^{(l)}$，状态转移定义为 ${\mathbf{M}}_{t+1,h}^{(l)}={\tilde{\mathbf{M}}}_{t,L_t,h}^{(l)}$，并且

$${\mathbf{D}}_{t,i,h}^{(l)}=\mathrm{Diag}[\exp \left({\mathit{\lambda }}_{t,i,h}^{(l)}\right)].$$

The detailed analysis of our stable mechanism on GDN-2 is presented in Appendix C.

关于我们在 GDN-2 上采用的稳定机制，详细分析见 附录 C。

Native Memory Utilization Procedure. In MemT-1.0, we remove the key-value normalization from Metis. Because the memory utilization operates at the token level in parallel, we omit the decoding index $k$ for simplicity. For the step $t$, we read out information from the current memory state ${\mathbf{M}}_{t,h}^{(l)}$ by

原生记忆利用过程。 在 MemT-1.0 中，我们移除了 Metis 中的键值归一化。 由于记忆利用在 token 级别并行进行，为简洁起见，我们省略解码索引 $k$。 对于步骤 $t$，我们通过下式从当前记忆状态 ${\mathbf{M}}_{t,h}^{(l)}$ 中读出信息：

$$\begin{array}{c}{\mathbf{R}}_{t,h}^{(l)}=\frac{{\mathbf{Q}}_{t,h}^{(l)}{\mathbf{M}}_{t,h}^{(l)}}{\sqrt{d_k}},\\ {\mathbf{Q}}_{t,h}^{(l)}=\mathrm{L}2\mathrm{Norm}\left({\tilde{\mathbf{H}}}_t^{(l)}{\tilde{\mathbf{W}}}_{Q,h}^{(l)}\right),\end{array}$$

where ${\tilde{\mathbf{W}}}_{Q,h}^{(l)}$ is the memory query projection matrix. After that, we calculate the memory output gate by

其中，${\tilde{\mathbf{W}}}_{Q,h}^{(l)}$ 为记忆查询投影矩阵。 之后，记忆输出门的计算为

$${\mathbf{G}}_{t,h}^{(l)}={\tilde{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_{g/d}^{(l)}{\mathbf{W}}_{g/u,h}^{(l)},$$

where ${\mathbf{W}}_{g/d}^{(l)},{\mathbf{W}}_{g/u,h}^{(l)}$ are parameters of the readout gate. Finally, the memory attention is calculated by

其中，${\mathbf{W}}_{g/d}^{(l)},{\mathbf{W}}_{g/u,h}^{(l)}$ 为读出门的参数。 最后，记忆注意力的计算为

$$\begin{array}{c}{\mathbf{A}}_{\text{mem},t}^{(l)}=\mathrm{MemoryAttention}({\tilde{\mathbf{H}}}_{t,k}^{(l)};{\mathbf{M}}_t^{(l)})\\ ={\mathrm{Concat}}_{h=1}^{H_q}\left[\begin{array}{c}\mathrm{RMSNorm}\left({\mathbf{R}}_{t,h}^{(l)}\right)\\ \odot \mathrm{SoftCapSiLU}\left({\mathbf{G}}_{t,h}^{(l)}\right)\end{array}\right],\end{array}$$

where

其中

$$\begin{array}{c}\mathrm{SoftCapSiLU}(x)=\tau \tanh \left(\frac{\mathrm{SiLU}(x)}{\tau }\right),\\ \mathrm{SiLU}(x)=x\sigma (x),\end{array}$$

and $\tau >0$ controls the soft-cap threshold, with $\tau =2$ by default. Target layers, projection strategies, and other settings vary slightly to accommodate different backbones, and more details can be found in Appendix B.

$\tau >0$ 控制软截断阈值，默认取 $\tau =2$。 为适应不同骨干模型，目标层、投影策略和其他设置略有不同，更多细节见 附录 B。

2.3. Scaling Native Memory Capacity

图5：扩展原生记忆容量。MoS 将每个原生记忆块扩展为多个记忆专家，并在共享的专家地址上分别执行稠密的存储路由与利用路由。

To jointly scale the online memory capacity and the trainable memory parameters, we introduce Mixture-of-Storage (MoS), as presented in Figure 5, which expands a native memory block into $E$ parallel memory experts. Unlike conventional sparse Mixture-of-Experts (MoE), MoS adopts dense routing over all memory experts, so every expert participates in the memory computation and the routing probabilities determine how information is read from and written into different memory experts.

为了同时扩展在线记忆容量和可训练记忆参数，我们引入混合存储（MoS），如 图5 所示，将一个原生记忆块扩展为 $E$ 个并行记忆专家。 与常规稀疏混合专家（MoE）不同，MoS 对所有记忆专家采用稠密路由，因此每个专家都参与记忆计算，路由概率决定如何向不同记忆专家写入信息，以及从中读取信息。

Memory Expert Expansion. For the $l$-th MoS layer, we extend the local memory state with an additional expert dimension as

记忆专家扩展。 对于第 $l$ 个 MoS 层，我们增加一个专家维度，将局部记忆状态扩展为

$${\mathcal{M}}_{t,\mathrm{MoS}}^{(l)}={\left\{{\mathbf{M}}_{t,e}^{(l)}\right\}}_{e=1}^E\in {\mathbb{R}}^{E\times H_{kv}\times d_k\times d_v},$$

where ${\mathbf{M}}_{t,e}^{(l)}\in {\mathbb{R}}^{H_{kv}\times d_k\times d_v}$ is the dynamic memory state of the $e$-th expert, and each expert has independent hyper memory parameters. All experts and both routers receive the same pre-attention hidden states ${\tilde{\mathbf{H}}}_t^{(l)}$.

其中，${\mathbf{M}}_{t,e}^{(l)}\in {\mathbb{R}}^{H_{kv}\times d_k\times d_v}$ 为第 $e$ 个专家的动态记忆状态，每个专家都有独立的超记忆参数。 所有专家及两个路由器都接收相同的注意力前置隐藏状态 ${\tilde{\mathbf{H}}}_t^{(l)}$。

Shared-Address Routing. MoS introduces two dense routers to control memory utilization and memory storage separately, which share a trainable expert address matrix

共享地址路由。 MoS 引入两个稠密路由器，分别控制记忆利用和记忆存储，并共享一个可训练的专家地址矩阵

$${\mathbf{C}}^{(l)}={[{\mathbf{c}}_1^{(l)},{\mathbf{c}}_2^{(l)},\cdots ,{\mathbf{c}}_E^{(l)}]}^{\mathrm{\top }}\in {\mathbb{R}}^{E\times d_{\mathrm{addr}}},$$

where $d_{\mathrm{addr}}$ is the dimension of the address representation. Then, we use router projections ${\mathbf{W}}_{\text{route/u}}^{(l)},{\mathbf{W}}_{\text{route/s}}^{(l)}\in {\mathbb{R}}^{d\times d_{\mathrm{addr}}}$ for memory utilization and storage separately. After that, routing probabilities of memory utilization are calculated by

其中，$d_{\mathrm{addr}}$ 为地址表示的维度。 然后，我们分别使用路由投影 ${\mathbf{W}}_{\text{route/u}}^{(l)},{\mathbf{W}}_{\text{route/s}}^{(l)}\in {\mathbb{R}}^{d\times d_{\mathrm{addr}}}$ 进行记忆利用和存储。 随后，记忆利用的路由概率计算如下：

$${\mathit{\alpha }}_{\text{u},t}^{(l)}=\mathrm{Softmax}\left({\tilde{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_{\text{route/u}}^{(l)}{\mathbf{C}}^{(l)\mathrm{\top }}\right)\in {\mathbb{R}}^{L_t\times E}.$$

Similarly, routing probabilities of memory storage are obtained by

类似地，记忆存储的路由概率为

$${\mathit{\alpha }}_{\text{s},t}^{(l)}=\mathrm{Softmax}\left({\tilde{\mathbf{H}}}_t^{(l)}{\mathbf{W}}_{\text{route/s}}^{(l)}{\mathbf{C}}^{(l)\mathrm{\top }}\right)\in {\mathbb{R}}^{L_t\times E}.$$

Both softmax operations are applied along the expert dimension. Therefore, the utilization and storage routers share the same semantic expert addresses but learn different projections.

两个 softmax 操作都沿专家维度进行。 因此，利用路由器和存储路由器共享相同的语义专家地址，但学习不同的投影。

Mixture-of-Storage Fusion. For memory utilization, each expert independently follows the native memory utilization procedure described above. We denote the memory-attention output of the $e$-th memory expert by

混合存储融合。 对于记忆利用，每个专家独立执行上述原生记忆利用过程。 我们将第 $e$ 个记忆专家的记忆注意力输出记为

$${\mathbf{A}}_{\mathrm{mem},t,e}^{(l)}={\mathrm{MemoryAttention}}_e({\tilde{\mathbf{H}}}_t^{(l)};{\mathbf{M}}_{t,e}^{(l)}).$$

The final memory attention at the $i$-th token in sequence $S_t$ is obtained by

序列 $S_t$ 中第 $i$ 个 token 的最终记忆注意力通过下式获得：

$${\mathbf{A}}_{\mathrm{mem},t,i}^{(l)}=\sum _{e=1}^E{\mathit{\alpha }}_{\text{u},t,i,e}^{(l)}{\mathbf{A}}_{\mathrm{mem},t,i,e}^{(l)}.$$

For memory storage, each expert independently produces its memory key, value, erase gate, write gate, and log-decay using its expert-specific parameters. For the $e$-th expert, the erase gate, write gate, and log-decay are calculated by

对于记忆存储，每个专家使用自身专属参数，独立生成记忆键、值、擦除门、写入门和对数衰减。 对于第 $e$ 个专家，擦除门、写入门和对数衰减计算如下：

$$\begin{array}{c}{\tilde{\mathit{\beta }}}_{t,e}^{(l)}={\mathit{\alpha }}_{\text{s},t,e}^{(l)}\odot {\mathit{\beta }}_{t,e}^{(l)},\\ {\tilde{\mathit{\omega }}}_{t,e}^{(l)}={\mathit{\alpha }}_{\text{s},t,e}^{(l)}\odot {\mathit{\omega }}_{t,e}^{(l)},\\ {\tilde{\mathit{\lambda }}}_{t,e}^{(l)}={\mathit{\lambda }}_{t,e}^{(l)}.\end{array}$$

After that, we calculate

之后，我们计算

$${\mathbf{U}}_{t,e}^{(l)}=\sqrt{{\tilde{\mathit{\beta }}}_{t,e}^{(l)}}\odot {\mathbf{K}}_{t,e}^{(l)}.$$

Finally, the memory state can be updated by

最后，记忆状态可更新为

$$\begin{array}{c}{\tilde{\mathbf{M}}}_{t,i,e,h}^{(l)}=\\ [\mathbf{I}-{\mathbf{U}}_{t,i,e,h}^{(l)\mathrm{\top }}{\mathbf{U}}_{t,i,e,h}^{(l)}]\cdot {\tilde{\mathbf{D}}}_{t,i,e,h}^{(l)}{\tilde{\mathbf{M}}}_{t,i-1,e,h}^{(l)}\\ +{\mathbf{U}}_{t,i,e,h}^{(l)\mathrm{\top }}({\tilde{\mathit{\omega }}}_{t,i,e,h}^{(l)}\odot {\mathbf{V}}_{t,i,e,h}^{(l)}),\end{array}$$

where we have the initialization ${\tilde{\mathbf{M}}}_{t,0,e,h}^{(l)}={\mathbf{M}}_{t,e,h}^{(l)}$, and transformation ${\mathbf{M}}_{t+1,e,h}^{(l)}={\tilde{\mathbf{M}}}_{t,L_t,e,h}^{(l)}$, with

其中，初始化为 ${\tilde{\mathbf{M}}}_{t,0,e,h}^{(l)}={\mathbf{M}}_{t,e,h}^{(l)}$，状态转换为 ${\mathbf{M}}_{t+1,e,h}^{(l)}={\tilde{\mathbf{M}}}_{t,L_t,e,h}^{(l)}$，并且

$${\tilde{\mathbf{D}}}_{t,i,e,h}^{(l)}=\mathrm{Diag}[\exp \left({\tilde{\mathit{\lambda }}}_{t,i,e,h}^{(l)}\right)].$$

3. Training

3.1. Backbone Preparation

A memory foundation model inherits from its backbone both the knowledge fixed at pre-training and the reliability with which that knowledge becomes behavior, and post-training treats the two very differently. Training on the backbone's own verified samples fits its distribution conditioned on verifier acceptance: it reweights what the backbone already produces and, by itself, leaves tasks the backbone almost never solves untouched. Our per-item analysis bears this out. Gains concentrate on tasks the backbone solves occasionally. Tasks it never solved in eight attempts end at only 1.6–4.0% accuracy across four benchmarks after the preparation described below, in line with observations for RL, whereas on a separate probe set of hard knowledge questions, supplying the relevant facts in context raises accuracy from 27% to 91%, which suggests that the remaining gap is largely informational. In our setting, reliability is therefore what post-training mainly improves, and new information is what native memory must supply.

记忆基础模型既继承骨干在预训练中固定下来的知识，也继承将这些知识转化为行为时的可靠性，而后训练对这两者的作用截然不同。 使用骨干自身通过验证的样本进行训练，拟合的是以验证器接受为条件的骨干分布：它重新调整骨干已有输出的权重，本身却无法改善那些骨干几乎从来不能解决的任务。 逐题分析证实了这一点。 收益主要集中在骨干偶尔能够解决的任务上。 对于八次尝试均未成功的任务，经过下述准备后，在四个基准上的准确率仍仅为 1.6–4.0%，与强化学习中的观察一致；而在另一组困难知识问题的探测集上，将相关事实放入上下文后，准确率从 27% 升至 91%，表明剩余差距主要与信息有关。 因此，在我们的设置中，后训练主要改善可靠性，而新信息则应由原生记忆提供。

Accordingly, we post-train DeepSeek-V4-Flash before attaching native memory, with data from a three-step pipeline applied within each of three tracks: general capabilities (instruction following, knowledge, mathematics, and long-context reasoning), agentic coding, and agentic working (tool use and workflow automation). First, capability profiling: each task is attempted repeatedly and classified as reliably, occasionally, or not solved, a profile that stands in for the pre-training data we cannot see. Second, attribution by intervention: a failure is re-run with the suspected missing piece, a grader convention, a procedural hint, or a fact, placed in context, and whether it disappears tells a missing convention or procedure apart from missing knowledge; failures that persist under every intervention are excluded from synthesis. Third, capability-conditioned synthesis assigns each class its own data source.

据此，我们在接入原生记忆之前先对 DeepSeek-V4-Flash 进行后训练，数据来自三个方向各自采用的三步流程：通用能力（指令遵循、知识、数学和长上下文推理）、智能体编程，以及智能体办公（工具使用和工作流自动化）。 第一步是能力画像：对每项任务重复尝试，并将其分类为稳定解决、偶尔解决或无法解决，以此代替我们无法查看的预训练数据。 第二步是干预归因：将怀疑缺失的部分——评分器约定、过程提示或事实——放入上下文后重新执行失败任务；失败是否消失，可用于区分缺少约定或过程知识与缺少事实知识；对所有干预均无响应的失败不纳入合成。 第三步是按能力条件进行合成，为每一类任务分配各自的数据来源。

Reliably solved tasks supply replay. Occasionally solved tasks supply the backbone's own attempts that pass programmatic verifiers (constraint checkers, unit tests, or sandboxed agent runs), which keeps most of the data on the backbone's own distribution. Failures traced to a missing convention or procedure are learned either from guided sampling, where the guidance appears in the sampling context but not in the target, or from style-matched demonstrations by stronger models. Knowledge gaps, which synthesis has so far closed only item by item, are left mainly to retrieval and, in MemT-1.0, to native memory.

能够稳定解决的任务提供回放数据。 偶尔解决的任务提供骨干自身通过程序化验证器的尝试（如约束检查器、单元测试或沙箱中的智能体运行），从而使大部分数据保持在骨干自身的分布内。 对于归因于缺少约定或过程知识的失败，学习数据来自引导采样——引导只出现在采样上下文中，不进入目标——或更强模型提供的风格匹配示范。 知识缺口主要留给检索，以及 MemT-1.0 中的原生记忆来填补；迄今为止，合成只能逐题弥补这些缺口。

Transfer is checked on held-out task families.

我们在留出的任务家族上检查迁移效果。

The recipe was settled over five rounds of full-mixture training and more than fifty training runs in total. The final mixture holds 126.5K trajectories (2.4B tokens, 1.1B loss-bearing), 56.4% general, 18.0% agentic coding, and 25.3% agentic working by count; agentic trajectories account for about 90% of loss-bearing tokens, so general data anchors the backbone's existing capabilities while the long agentic trajectories carry the multi-turn behavior that memory tasks build on. The prepared backbone improves on all six benchmarks we monitor across the three tracks, by 6.3 points on average over the public release (Figure 2), and on SWE-bench Multilingual and IFBench v2, at 284B total and 13B active parameters, it exceeds the reported scores of larger open models, including GLM-5.3 Flash (320B total, 18B active) and the far larger DeepSeek-V4 Pro (1.6T total, 49B active).

最终方案经过五轮全混合数据训练、累计超过五十次训练运行才确定。 最终混合数据包含 126.5K 条轨迹（2.4B token，其中 1.1B token 计入损失）；按条数计，通用、智能体编程和智能体办公分别占 56.4%、18.0% 和 25.3%；智能体轨迹约占计入损失 token 的 90%，因此通用数据用于稳固骨干的现有能力，而长智能体轨迹承载记忆任务所依赖的多轮行为。 准备后的骨干在三个方向上监测的六个基准中均有提升，相较公开版本平均提高 6.3 个点（图2）；在 SWE-bench Multilingual 和 IFBench v2 上，它以总计 284B、激活 13B 的参数规模，超过了更大开放模型所报告的成绩，包括 GLM-5.3 Flash（总计 320B、激活 18B），以及大得多的 DeepSeek-V4 Pro（总计 1.6T、激活 49B）。

The backbone is then frozen to serve as both the host model and the full-context distillation teacher of Section 3.3. Backbone preparation and native memory are two routes by which experience shapes a model's later behavior, through the weights before deployment and through memory states during interaction; closing the loop between them is left to future work.

随后冻结骨干，使其同时作为宿主模型，以及 第 3.3 节 中的完整上下文蒸馏教师。 骨干准备和原生记忆是经验塑造模型后续行为的两条途径，分别通过部署前的权重与交互中的记忆状态实现；如何将二者闭环连接，留待未来工作。

3.2. Data Construction

To support the training of MemT-1.0, we construct a large-scale training corpus with 34B tokens, which is composed of continued pre-training (CPT) and post-training data. We collect 20B tokens of CPT data, which mainly consists of large-scale general-purpose language modeling corpora. It contains long-form and sequential text to expose the model to diverse linguistic patterns and extensive contextual dependencies.

为支持 MemT-1.0 的训练，我们构建了包含 34B token 的大规模训练语料，由持续预训练（CPT）数据和后训练数据组成。 我们收集了 20B token 的 CPT 数据，主要由大规模通用语言建模语料构成。 其中包含长篇和连续文本，使模型接触多样的语言模式及广泛的上下文依赖。

For post-training, we organize 14B tokens of data into three major categories. First, we construct memory-native language modeling data. This data encourages MemT-1.0 to maintain linguistic coherence, stylistic consistency, and instruction-following capability when information is accessed through native memory. Second, we construct data for native memory storage and utilization, including tasks such as summarization, continuation, and compression.

对于后训练，我们将 14B token 的数据组织为三大类。 首先，构建记忆原生语言建模数据。 这些数据鼓励 MemT-1.0 在通过原生记忆访问信息时，保持语言连贯性、风格一致性和指令遵循能力。 其次，构建原生记忆存储与利用数据，包括摘要、续写和压缩等任务。

They provide explicit supervision for storing information that should be retained in native memory. They train the model to recover and utilize the stored information during subsequent generation. Third, we introduce memory task data to further strengthen the model's long-range memory capability. They further improve the model's ability to selectively retain critical information and resolve long-range dependencies, which improves the utilization of memory when answering downstream queries.

这些任务为如何存储应当保留在原生记忆中的信息提供显式监督。 它们训练模型在后续生成中恢复并利用已存储的信息。 第三，引入记忆任务数据，进一步增强模型的长程记忆能力。 这些任务进一步提升模型有选择地保留关键信息、解决长程依赖的能力，从而改善回答下游查询时对记忆的利用。

Part of the training corpus is based on real-world interaction patterns in MemOS's anonymized internal data.

部分训练语料基于 MemOS 匿名化内部数据中的真实交互模式。

All three categories are formulated in a unified context-query-answer format. The context is first processed and stored through the native memory mechanism, after which the model receives the corresponding query and is trained to generate the target answer conditioned on the resulting memory state. This unified formulation allows heterogeneous memory-related tasks to be incorporated into the same post-training pipeline.

这三类任务均采用统一的上下文—查询—答案格式。 首先通过原生记忆机制处理并存储上下文，之后模型接收相应查询，并学习以得到的记忆状态为条件生成目标答案。 这一统一形式使异构的记忆相关任务能够纳入同一后训练流程。

3.3. Training Objectives

The core training objective of MemT-1.0 is to align its capability with the backbone given full context, which enables the model to retain and utilize historical information without preserving the context. To this end, we introduce an off-policy distillation framework for training native memory models, where the backbone serves as the teacher model for acquiring memory capabilities.

MemT-1.0 的核心训练目标是，使其能力对齐到获得完整上下文的骨干，从而在不保留上下文的情况下，仍能保留并利用历史信息。 为此，我们引入用于训练原生记忆模型的离策略蒸馏框架，由骨干担任教师模型，帮助学生获取记忆能力。

Let ${\theta }_b$ denote the frozen backbone parameters and $\varphi$ denote the trainable memory parameters, including the hyper memory blocks and extra parameters of MoS. For a training trajectory $\{(X_t,Y_t){\}}_{t=1}^T$, we define the preceding textual history as $C_{<t}=S_1\Vert \cdots \Vert S_{t-1}$, where $S_t=X_t\Vert Y_t$. We use the original backbone without the additional memory modules as the teacher, whose next-token distribution is

令 ${\theta }_b$ 表示冻结的骨干参数，$\varphi$ 表示可训练记忆参数，包括超记忆块和 MoS 的额外参数。 对于训练轨迹 $\{(X_t,Y_t){\}}_{t=1}^T$，我们将此前的文本历史定义为 $C_{<t}=S_1\Vert \cdots \Vert S_{t-1}$，其中 $S_t=X_t\Vert Y_t$。 我们使用未增加记忆模块的原始骨干作为教师，其下一 token 分布为

$$\begin{array}{c}{q}_{t,k}(v)=P(v\mid C_{<t},X_t,Y_{t,<k};{\theta }_b),\\ v\in \mathcal{V}.\end{array}$$

The MemT-1.0 student receives the current input and the same observed response prefix, with previous steps represented by the incoming local memory state ${\mathbf{M}}_t$, where

MemT-1.0 学生接收当前输入和相同的已观测回复前缀，此前步骤则由传入的局部记忆状态 ${\mathbf{M}}_t$ 表示，其中

$$p_{t,k}(v)=P(v\mid X_t,Y_{t,<k};{\theta }_b,\varphi ,{\mathcal{M}}_t).$$

Here, ${\mathcal{M}}_t$ denotes the collection of dynamic states across all memory layers and experts. The state is initialized to zero for each independent trajectory and evolves through the native storage procedure after each completed step. Both models are evaluated under teacher forcing on fixed teacher-generated trajectories rather than student-generated rollouts, making the distillation objective off-policy with respect to the student policy. To reduce the size of the teacher targets, we retain its Top-$K$ tokens at each supervised position. Let ${\mathcal{K}}_{t,k}\subseteq \mathcal{V}$ be this token set and $T_d>0$ be the distillation temperature. Given $q^{T_d}$ and $p^{T_d}$ for distributions obtained by dividing the respective logits by $T_d$ before softmax, we define

这里，${\mathcal{M}}_t$ 表示所有记忆层及专家的动态状态集合。 每条独立轨迹的状态都初始化为零，并在每个完整步骤结束后通过原生存储过程演化。 两个模型都在固定的教师生成轨迹上，采用教师强制进行评估，而非使用学生生成的 rollout，因此相对于学生策略，蒸馏目标属于离策略目标。 为缩小教师目标的规模，我们在每个受监督位置仅保留教师概率最高的 $K$ 个 token。 令 ${\mathcal{K}}_{t,k}\subseteq \mathcal{V}$ 为这一 token 集合，$T_d>0$ 为蒸馏温度。 对于将各自 logits 除以 $T_d$ 后再进行 softmax 所得到的分布 $q^{T_d}$ 和 $p^{T_d}$，我们定义

$$\begin{array}{c}{\hat{q}}_{t,k}(v)=\frac{q_{t,k}^{T_d}(v)}{\sum _{u\in {\mathcal{K}}_{t,k}}{q}_{t,k}^{T_d}(u)},\\ v\in {\mathcal{K}}_{t,k},\end{array}$$

and set ${\hat{q}}_{t,k}(v)=0$ outside this set. The per-token distillation loss is the forward KL divergence from this truncated teacher distribution to the student by

并在该集合之外令 ${\hat{q}}_{t,k}(v)=0$。 逐 token 蒸馏损失为从截断教师分布到学生分布的前向 KL 散度：

$${\ell }_{t,k}^{\mathrm{KL}}=T_d^2\sum _{v\in {\mathcal{K}}_{t,k}}{\hat{q}}_{t,k}(v)\log \frac{{\hat{q}}_{t,k}(v)}{p_{t,k}^{T_d}(v)}.$$

The teacher is renormalized over its selected support, whereas the student remains normalized over the full vocabulary. Consequently, the loss also penalizes student probability mass assigned outside the teacher's selected tokens. We additionally use the observed target token $y_{t,k}$ for next-token cross-entropy supervision

教师在选定的支持集上重新归一化，而学生仍在整个词表上归一化。 因此，损失也会惩罚学生分配到教师选定 token 之外的概率质量。 此外，我们使用已观测的目标 token $y_{t,k}$，进行下一 token 交叉熵监督：

$${\ell }_{t,k}^{\mathrm{CE}}=-\log p_{t,k}(y_{t,k}).$$

For a training batch $\mathcal{B}$, let $m_{t,k}^{(b)}\in \{0,1\}$ indicate whether a position in trajectory $b$ is supervised. The training loss is

对于训练批次 $\mathcal{B}$，令 $m_{t,k}^{(b)}\in \{0,1\}$ 表示轨迹 $b$ 中某个位置是否接受监督。 训练损失为

$$\begin{array}{c}\mathcal{L}(\varphi )=\\ \frac{1}{N_{\mathrm{spt}}}\sum _{b\in \mathcal{B}}\sum _{t,k}{m}_{t,k}^{(b)}[{\lambda }_{\mathrm{KD}}{\ell }_{t,k}^{\mathrm{KL},(b)}+{\lambda }_{\mathrm{CE}}{\ell }_{t,k}^{\mathrm{CE},(b)}],\end{array}$$

where $N_{\mathrm{spt}}=\sum _{b\in \mathcal{B}}\sum _{t,k}{m}_{t,k}^{(b)}$ is the number of supervised tokens, and ${\lambda }_{\mathrm{KD}}$ and ${\lambda }_{\mathrm{CE}}$ are nonnegative weights with at least one nonzero value. Cross-entropy uses the untempered student distribution. The mask selects the prediction targets appropriate to each training stage and excludes padding. History tokens may participate in memory construction without receiving direct loss.

其中，$N_{\mathrm{spt}}=\sum _{b\in \mathcal{B}}\sum _{t,k}{m}_{t,k}^{(b)}$ 为受监督 token 的数量，${\lambda }_{\mathrm{KD}}$ 和 ${\lambda }_{\mathrm{CE}}$ 为非负权重，且至少有一个非零。 交叉熵使用未经温度调整的学生分布。 掩码选择适合各训练阶段的预测目标，并排除填充位置。 历史 token 可以参与记忆构建，而不直接计入损失。

Normalization is performed over all supervised tokens in the optimization batch, including those distributed across workers and accumulation steps. Specifically, we use CE supervision during CPT and KL-based distillation during post-training. For MoS training, we add a router auxiliary loss to balance the load across memory experts. For MemT-1.0-Flash, the backbone prepared in Section 3.1 is frozen during native-memory training.

归一化基于优化批次中全部受监督 token 进行，包括分布在不同工作进程和梯度累积步骤中的 token。 具体来说，CPT 阶段使用 CE 监督，后训练阶段使用基于 KL 的蒸馏。 训练 MoS 时，我们加入路由器辅助损失，以平衡各记忆专家的负载。 对于 MemT-1.0-Flash，第 3.1 节 中准备好的骨干在原生记忆训练期间保持冻结。

3.4. Continued Pre-Training

We first conduct CPT to establish the basic capability of language modeling with native memory. This stage trains MemT-1.0 to predict subsequent tokens using both the current textual context and information accumulated from preceding segments. For each training sequence, we divide the tokens into consecutive, non-overlapping segments and treat each segment as a step $t$. The student performs causal language modeling within the current segment, with the preceding history $C_{<t}$ represented by the incoming memory states ${\mathcal{M}}_t$.

我们首先进行 CPT，建立利用原生记忆进行语言建模的基本能力。 这一阶段训练 MemT-1.0 同时利用当前文本上下文和前面片段积累的信息来预测后续 token。 对于每条训练序列，我们将 token 划分为连续、不重叠的片段，并把每个片段视为步骤 $t$。 学生在当前片段内进行因果语言建模，此前历史 $C_{<t}$ 由传入的记忆状态 ${\mathcal{M}}_t$ 表示。

Its backbone context is restricted to the current segment, so information from earlier segments is accessed through native memory. After processing a segment, the native storage procedure applies token-level updates to obtain ${\mathcal{M}}_{t+1}$ for the next segment. The final segment is retained even when it is shorter than the specified segment length. Memory states are carried across segments within the same sequence and initialized to zero at the beginning of each independent sequence.

骨干上下文被限制为当前片段，因此更早片段的信息通过原生记忆访问。 处理完一个片段后，原生存储过程应用 token 级更新，为下一个片段得到 ${\mathcal{M}}_{t+1}$。 即使最后一个片段短于指定片段长度，也予以保留。 记忆状态在同一序列的各片段间传递，并在每条独立序列开始时初始化为零。

We use CE loss during CPT, with the ground-truth next tokens serving as the supervision targets. The student performs causal language modeling within the current segment while relying on native memory to access information from preceding segments. This encourages the model to establish language modeling capabilities under native memory without requiring the full historical context as textual input.

CPT 阶段使用 CE 损失，以真实的下一 token 作为监督目标。 学生在当前片段内进行因果语言建模，同时依赖原生记忆访问前面片段的信息。 这鼓励模型在原生记忆条件下建立语言建模能力，而不要求以文本输入的形式提供完整历史上下文。

3.5. Post-Training

Post-training further adapts MemT-1.0 to native memory-based response generation. We initialize the memory parameters from the CPT checkpoint and train on instances containing historical context, queries, and reference responses. The historical context is processed chronologically through the native storage procedure, producing the memory states used to answer subsequent queries. The responses are rollouts from the full-context teacher, and serve to condition both models under teacher forcing rather than as direct supervision targets.

后训练进一步使 MemT-1.0 适应基于原生记忆的回复生成。 我们从 CPT 检查点初始化记忆参数，并在包含历史上下文、查询和参考回复的样本上训练。 历史上下文按照时间顺序通过原生存储过程处理，生成用于回答后续查询的记忆状态。 回复来自完整上下文教师的 rollout，在教师强制下作为两个模型的条件，而非直接监督目标。

At step $t$, the student receives the current query $X_t$ and the observed response prefix $Y_{t,<k}$, with the preceding history $C_{<t}$ represented by ${\mathcal{M}}_t$. The teacher retains access to the corresponding textual history. Memory states are initialized independently for each trajectory, preserving the association between a query and its own history.

在步骤 $t$，学生接收当前查询 $X_t$ 和已观测回复前缀 $Y_{t,<k}$，此前历史 $C_{<t}$ 则由 ${\mathcal{M}}_t$ 表示。 教师仍能访问对应的文本历史。 每条轨迹的记忆状态独立初始化，以保持查询与其自身历史的关联。

We use the KL loss in the post-training stage. The supervision mask $m_{t,k}^{(b)}$ selects response tokens, while history, query, and padding positions receive no direct token loss. Both teacher and student are evaluated under teacher forcing with the same observed response prefix. Reference responses thus determine the prefixes on which the models are evaluated, while the teacher's probability distributions provide the distillation targets.

后训练阶段使用 KL 损失。 监督掩码 $m_{t,k}^{(b)}$ 选择回复 token，历史、查询和填充位置不直接计入 token 损失。 教师与学生在相同的已观测回复前缀下，均采用教师强制进行评估。 因此，参考回复决定评估模型时所使用的前缀，而教师的概率分布提供蒸馏目标。

The loss is averaged over all supervised response tokens in the training batch. This focuses optimization on the outputs required by the task while training the student to approximate the full-context teacher through native memory. Although historical tokens receive no direct loss, gradients from response predictions propagate through memory readout and earlier storage operations. This allows the model to learn which historical information to retain and how to utilize it for the current query.

损失在训练批次中所有受监督回复 token 上取平均。 这使优化集中在任务所需输出上，同时训练学生通过原生记忆逼近完整上下文教师。 尽管历史 token 不直接计入损失，回复预测产生的梯度仍会经由记忆读出及更早的存储操作传播。 这使模型能够学习应保留哪些历史信息，以及如何利用这些信息回答当前查询。

The full-context teacher provides a reference for both the use of historical information and the backbone's response behavior, encouraging MemT-1.0 to acquire task-specific memory capabilities while preserving its general capabilities.

完整上下文教师同时为历史信息利用和骨干回复行为提供参考，促使 MemT-1.0 在保留通用能力的同时，获得面向具体任务的记忆能力。

3.6. Training Setups

For MemT-1.0-Nano, training consists of 1 CPT epoch followed by post-training. The global batch size is 64 packed sequences. Each pack contains at most 64K tokens, while the segment length is 2K tokens. Incomplete final global batches are dropped.

MemT-1.0-Nano 的训练包括 1 个 epoch 的 CPT，随后进行后训练。 全局批大小为 64 个打包序列。 每个 pack 最多包含 64K token，片段长度为 2K token。 不足一个完整全局批次的最后一批数据被丢弃。

For the distillation objective, we retain the teacher's top-$K$ tokens at each supervised position with $K=64$, and set the distillation temperature to $T_d=1.0$. We optimize only the memory parameters using AdamW with ${\beta }_1=0.9$, ${\beta }_2=0.999$, $ϵ={10}^{-8}$, weight decay $0.01$, and a gradient clipping norm of $1.0$. We use mixed-precision training. The backbone runs in BF16, while numerically sensitive components (e.g., RMSNorm, Softmax, and memory states) and optimizer states are kept in FP32.

对于蒸馏目标，我们在每个受监督位置保留教师概率最高的 $K$ 个 token，取 $K=64$，并设置蒸馏温度 $T_d=1.0$。 我们仅使用 AdamW 优化记忆参数，设置 ${\beta }_1=0.9$、${\beta }_2=0.999$、$ϵ={10}^{-8}$、权重衰减 $0.01$ 和梯度裁剪范数 $1.0$。 我们采用混合精度训练。 骨干以 BF16 运行，而对数值敏感的组件（如 RMSNorm、Softmax 和记忆状态）及优化器状态保持 FP32。

A soft cap is applied to the memory output gate, with a limit of 2.0, for training stability. We use a WSD learning rate scheduler for MemT-1.0-Nano training. During CPT, we warm up for 600 steps to a learning rate of $1\times {10}^{-4}$, and keep the peak rate constant for the remaining steps. During post-training, we warm up for 500 steps to $1\times {10}^{-4}$, keep it constant, and apply cosine decay over the final 1,000 steps to a minimum learning rate of $5\times {10}^{-6}$.

为保证训练稳定性，对记忆输出门应用上限为 2.0 的软截断。 MemT-1.0-Nano 的训练采用 WSD 学习率调度器。 在 CPT 阶段，用 600 步预热至学习率 $1\times {10}^{-4}$，之后各步保持峰值学习率不变。 在后训练阶段，用 500 步预热至 $1\times {10}^{-4}$，随后保持不变，并在最后 1,000 步进行余弦衰减，降至最低学习率 $5\times {10}^{-6}$。

For MemT-1.0-Flash, we initialize from the prepared backbone and use a global batch of 192 packs, with the same 64K maximum pack length and 2K-token storage steps. Memory training uses the online backbone teacher with Top-64 targets, temperature 1, and KL loss. The learning-rate schedule has a peak of $8\times {10}^{-5}$, a 100-step warmup, and a planned cosine decay over the final steps.

对于 MemT-1.0-Flash，我们从准备好的骨干初始化，采用 192 个 pack 的全局批次，同样使用 64K 的最大 pack 长度和 2K token 的存储步骤。 记忆训练使用在线骨干教师、Top-64 目标、温度 1 和 KL 损失。 学习率计划的峰值为 $8\times {10}^{-5}$，预热 100 步，并计划在最后若干步进行余弦衰减。

4. System and Infrastructure

Each interaction in MemT-1.0 depends on memory accumulated from earlier steps. Training must combine these dependencies with batched GPU computation, while inference must maintain an independent evolving state for each trajectory. Our infrastructure executes the model one decoder layer at a time, handles state recurrence within memory blocks, and manages model weights separately from trajectory-specific states. We describe the resulting training schedule, memory operators, and online state organization below.

MemT-1.0 的每次交互都依赖前面步骤积累的记忆。 训练必须将这些依赖与 GPU 批量计算结合起来，而推理必须为每条轨迹维护独立、不断演化的状态。 我们的基础设施逐个解码器层执行模型，在记忆块内部处理状态递推，并将模型权重与轨迹专属状态分开管理。 下面介绍由此形成的训练调度、记忆算子和在线状态组织方式。

4.1. Distributed Training

Training packs combine independent trajectories, each containing the steps defined in Section 3. Attention stays within each step, while memory carries information between steps of the same trajectory and resets at trajectory boundaries. Teacher forcing supplies all tokens in the pack, so the model processes the complete pack one decoder layer at a time. Within each layer, attention and memory projections are batched across tokens. The memory block alone advances state along each trajectory. This organization retains large backbone matrix multiplications while preserving the order of memory writes and reads.

训练 pack 组合多条独立轨迹，每条轨迹包含 第 3 节 定义的步骤。 注意力限制在各步骤内，记忆则在同一轨迹的步骤之间传递信息，并在轨迹边界重置。 教师强制提供 pack 中的全部 token，因此模型能够逐个解码器层处理完整 pack。 在每一层内，注意力和记忆投影跨 token 批量执行。 只有记忆块沿各条轨迹推进状态。 这一组织方式在保留骨干大矩阵乘法的同时，维持了记忆读写顺序。

We use Megatron-Core and Bridge for distributed training, including expert parallelism for the backbone MoE layers. Although backbone weights are frozen during memory training, gradients still pass through backbone computations and memory transitions, allowing later predictions to train earlier writes. The remaining infrastructure challenge is therefore the cost of memory computation and its training activations, which we address below.

我们使用 Megatron-Core 和 Bridge 进行分布式训练，包括骨干 MoE 层的专家并行。 尽管记忆训练期间骨干权重被冻结，梯度仍会通过骨干计算和记忆状态转移传播，使较晚的预测能够训练较早的写入。 因此，剩余的基础设施挑战是记忆计算及其训练激活值的成本，我们将在下文处理。

4.2. Efficient Operators

Native memory introduces two related costs: ordered state updates and activations that grow with the number of memory experts. Our execution separates state updates from reads of step snapshots, then groups complete steps into windows to limit the lifetime of expert projections. This lets the memory branch control its working set while the backbone continues to process the full token pack layer by layer.

原生记忆引入两个相关成本：有序状态更新，以及随记忆专家数量增长的激活值。 我们的执行方式将状态更新与步骤快照读取分离，再把完整步骤分组成窗口，以限制专家投影的存活时间。 这使记忆分支能够控制工作集规模，同时骨干继续逐层处理完整 token pack。

Chunked updates and snapshot reads. The stable update has a rank-one structure. Suppressing layer, step, and channel indices, let ${\mathbf{k}}_i$ denote the normalized key, and define ${\mathbf{u}}_i=\sqrt{{\mathit{\beta }}_i}\odot {\mathbf{k}}_i$, ${\mathbf{z}}_i={\mathit{\omega }}_i\odot {\mathbf{v}}_i$, and ${\mathbf{D}}_i=\mathrm{Diag}(\exp ({\mathit{\lambda }}_i))$. The update in Section 2 can then be written as

分块更新与快照读取。 稳定更新具有秩一结构。 省略层、步骤和通道索引，令 ${\mathbf{k}}_i$ 表示归一化键，并定义 ${\mathbf{u}}_i=\sqrt{{\mathit{\beta }}_i}\odot {\mathbf{k}}_i$、${\mathbf{z}}_i={\mathit{\omega }}_i\odot {\mathbf{v}}_i$ 和 ${\mathbf{D}}_i=\mathrm{Diag}(\exp ({\mathit{\lambda }}_i))$。 则 第 2 节 的更新可写为

$$\begin{array}{c}{\mathbf{S}}_i={\mathbf{D}}_i{\mathbf{S}}_{i-1}\\ +{\mathbf{u}}_i^{\mathrm{\top }}({\mathbf{z}}_i-{\mathbf{u}}_i{\mathbf{D}}_i{\mathbf{S}}_{i-1}),\\ {\mathbf{S}}_0={\mathbf{M}}_t,{\mathbf{M}}_{t+1}={\mathbf{S}}_{L_t}.\end{array}$$

Each token requires a vector–matrix multiplication and an outer product, with leading work of $O(d_k{d}_v)$ per channel. We use chunked WY to group these small dependent updates into triangular solves and matrix multiplications, carrying state between chunks and reconstructing intermediate quantities during backward. The adaptation to MemT-1.0 is to pair this writer with a separate snapshot reader: all queries in step $t$ read the same incoming state ${\mathbf{M}}_t$, even as the writer processes several chunks to form ${\mathbf{M}}_{t+1}$. The reads can therefore be batched independently, followed by RMSNorm and SoftCapSiLU output gating. Gradients from later reads propagate through the writes that produced their snapshots. Writer chunks determine computation granularity, and model steps determine when updated memory becomes visible.

每个 token 需要一次向量—矩阵乘法和一次外积，每个通道的主要计算量为 $O(d_k{d}_v)$。 我们使用分块 WY，将这些规模较小且相互依赖的更新组织为三角求解和矩阵乘法，在分块之间传递状态，并在反向传播时重构中间量。 针对 MemT-1.0 的适配，是为该写入器配备独立的快照读取器：步骤 $t$ 中所有查询都读取相同的传入状态 ${\mathbf{M}}_t$，即使写入器需要处理多个分块才能形成 ${\mathbf{M}}_{t+1}$。 因此，读取可独立批量执行，随后进行 RMSNorm 和 SoftCapSiLU 输出门控。 较晚读取的梯度会通过生成其快照的写入操作传播。 写入器分块决定计算粒度，而模型步骤决定更新后的记忆何时可见。

Complete-step windows for MoS. Dense MoS evaluates every expert, so materializing all expert projections at once requires tensors that grow with both pack length and expert count. Whole-pack execution does exactly this: it projects every token in the pack before processing the resulting memory computations. Windowed execution instead performs expert projections, state updates, and readout fusion for one group of complete steps at a time, passing a differentiable FP32 final state to the next window. Both schedules use the same storage and utilization routers, shared expert addresses, and update kernel. Storage probabilities modulate the erase gate, write gate, and log-decay. Utilization probabilities mix the gated expert readouts.

MoS 的完整步骤窗口。 稠密 MoS 会计算每个专家，因此一次性物化全部专家投影，需要的张量规模会同时随 pack 长度和专家数量增长。 整包执行正是如此：它先对 pack 中每个 token 做投影，再处理由此产生的记忆计算。 窗口化执行则每次只对一组完整步骤进行专家投影、状态更新和读出融合，并将可微的 FP32 最终状态传给下一个窗口。 两种调度使用相同的存储路由器、利用路由器、共享专家地址和更新内核。 存储概率调节擦除门、写入门和对数衰减。 利用概率则混合经过门控的专家读出。

A window budget limits how many tokens are projected together, while keeping each step intact even if it exceeds the budget. Recomputing internal intermediates during backward avoids retaining all windows' expert projections for the full training pass. Smaller windows reduce live intermediates but require more launches and recomputation. Outputs, routing probabilities, and boundary states still occupy memory, so the total remains dependent on pack length as well as window size.

窗口预算限制一次共同投影的 token 数量，同时保持每个步骤完整，即便某一步超过预算也不拆分。 在反向传播时重新计算内部中间量，可以避免在整个训练过程中保留所有窗口的专家投影。 更小的窗口减少活跃中间量，但需要更多启动和重计算。 输出、路由概率和边界状态仍占用内存，因此总占用仍同时依赖 pack 长度和窗口大小。

The memory–time tradeoff of window size. We measure one complete MoS memory block on a single GPU using synthetic hidden-state inputs. Table 1 fixes a 32K-token pack, BF16 parameters and projections, and 2048-token steps, comparing peak allocated memory and forward-plus-backward (F+B) time for two window sizes. Arrows denote whole-pack execution $\to$ windowed execution; all values are medians.

窗口大小的内存—时间权衡。 我们使用合成隐藏状态输入，在单个 GPU 上测量一个完整 MoS 记忆块。 表1 固定采用 32K token 的 pack、BF16 参数与投影，以及 2048 token 的步骤，比较两种窗口大小的峰值分配内存和前向加反向传播（F+B）时间。 箭头表示整包执行 $\to$ 窗口化执行；所有数值均为中位数。

表1：单个 GPU 上、32K token pack 长度下，一个 MoS 记忆块的内存—时间权衡。箭头表示从整包执行变为窗口化执行。数值均为中位数。

Window size (complete steps)	Peak memory (GiB)	Memory reduction	F+B (ms)	Time increase
2048 tokens (1 step)	47.83 → 6.03	87.4%	462.7 → 493.5	6.6%
4096 tokens (2 steps)	47.83 → 9.49	80.1%	463.0 → 470.6	1.6%

In this configuration, processing two steps at a time reduces peak memory by approximately 80%, with a 1.6% increase in median F+B time. Shrinking the window to one step saves additional memory but raises the time increase to 6.6%. The window budget thus provides a direct resource tradeoff: smaller windows reduce live intermediates, while larger windows reduce execution overhead. Appendix F provides the block configuration, measurement protocol, and full results and variability from 8K to 32K.

在此配置下，每次处理两个步骤可使峰值内存减少约 80%，F+B 时间中位数增加 1.6%。 将窗口缩小为一步，可进一步节省内存，但时间增幅会提高到 6.6%。 因此，窗口预算提供了直接的资源权衡：更小的窗口减少活跃中间量，更大的窗口减少执行开销。 附录 F 提供了记忆块配置、测量协议，以及 8K 到 32K 范围内的完整结果和波动情况。

4.3. Stateful Inference and Serving

During online interaction, backbone and hyper memory weights remain fixed and are shared across trajectories, while each trajectory maintains its own dynamic memory states. Within a step, generation reads the incoming memory snapshot, and attention over the current text uses that step's KV cache. Once the interaction completes, storage incorporates the input and response into memory for the next interaction. The serving runtime manages states by trajectory so that subsequent requests continue their respective histories.

在线交互期间，骨干与超记忆权重保持固定，并在各轨迹之间共享，而每条轨迹维护自身的动态记忆状态。 在单个步骤内，生成读取传入的记忆快照，当前文本上的注意力则使用该步骤的 KV 缓存。 交互完成后，存储过程将输入和回复纳入记忆，供下次交互使用。 服务运行时按轨迹管理状态，使后续请求能够延续各自的历史。

For a fixed configuration, the number of memory-state elements per trajectory is

对于固定配置，每条轨迹的记忆状态元素数量为

$$N_{\mathrm{state}}=\sum _l{E}_l{H}_{kv,l}{d}_{k,l}{d}_{v,l},$$

where the sum covers memory layers and $E_l=1$ without MoS. Growing history updates these elements while their number remains determined by configured memory capacity. Total storage for multiple trajectories grows with trajectory count. Active requests additionally require their current-step KV caches and computational workspaces. Persistent-state storage can thus be estimated directly from the memory configuration, whereas runtime GPU memory also includes shared weights and active requests. The effect of finite capacity on retained historical detail is examined in the capability evaluations and Discussion.

其中，求和遍历所有记忆层；不使用 MoS 时，$E_l=1$。 历史增长会更新这些元素，但其数量仍由配置的记忆容量决定。 多条轨迹的总存储量随轨迹数量增长。 活跃请求还需要各自当前步骤的 KV 缓存和计算工作区。 因此，持久状态存储量可以直接根据记忆配置估算，而运行时 GPU 内存还包括共享权重和活跃请求。 有限容量对保留历史细节的影响，将在能力评估和讨论中考察。

This organization separates history preparation from subsequent queries. In Evaluation's two-GPU, single-request BEAM experiment, MemT-1.0-Nano's query latency after history preparation remains nearly constant as history grows. At 64K tokens of history, MemT-1.0-Nano achieves an approximately 33$\times$ reduction in logical per-trajectory state size compared with full context. The storage metric measures the logical state size after history preparation and before querying, and excludes shared weights. When subsequent queries reuse that state, the initial preparation cost is amortized over multiple interactions. The full protocol and curves appear in Section 6 and Appendix D.5.

这一组织方式将历史准备与后续查询分离。 在评估部分使用双 GPU、单请求的 BEAM 实验中，MemT-1.0-Nano 在历史准备完成后的查询延迟，随着历史增长基本保持恒定。 当历史为 64K token 时，相较完整上下文，MemT-1.0-Nano 的每条轨迹逻辑状态大小约 缩减为原来的 1/33。 存储指标测量的是历史准备完成后、查询开始前的逻辑状态大小，不包括共享权重。 当后续查询复用该状态时，初始准备成本可分摊到多次交互中。 完整协议和曲线见 第 6 节 与 附录 D.5。

5. Memory Scaling Behavior

5.1. Overview

Scaling has been a key driver behind the continuous improvement of foundation models, where increasing model capacity, training data, and computation often leads to predictable gains in general capabilities. However, for memory foundation models, it remains unclear whether native memory capabilities exhibit similar scaling behavior. Exploring such scaling behavior is critical, as it reveals whether memory capability can continue to improve with scale and provides guidance for the design of larger memory foundation models. In particular, memory-related training and the capacity of native memory states are two important factors that may influence memory capability, motivating us to explore scaling behavior along these two aspects.

规模扩展一直是基础模型持续进步的关键驱动力；增加模型容量、训练数据和计算量，往往能够带来可预测的通用能力提升。 然而，对于记忆基础模型，原生记忆能力是否呈现类似的扩展行为，仍不清楚。 探索这一扩展行为至关重要，因为它能够揭示记忆能力能否随着规模扩大而持续提升，并为更大记忆基础模型的设计提供指导。 具体来说，记忆相关训练和原生记忆状态容量是可能影响记忆能力的两个重要因素，因此我们沿这两个方面探索扩展行为。

In this work, we study memory scaling along both dimensions. Data and training scaling explores how memory capability changes with continued exposure to memory-related post-training data, while memory capacity scaling studies the impact of increasing the capacity of native memory states. They allow us to characterize how native memory capability evolves with additional training and expanded memory capacity.

在本研究中，我们沿这两个维度研究记忆扩展。 数据与训练扩展考察持续接触记忆相关后训练数据时记忆能力如何变化，记忆容量扩展则研究增大原生记忆状态容量的影响。 二者使我们能够刻画原生记忆能力如何随额外训练和容量扩展而演变。

5.2. Scaling Protocol

We conduct all scaling experiments based on MemT-1.0-Nano under a controlled experimental protocol. We explore two scaling aspects, including data and training scaling and memory capacity scaling. The backbone architecture, training objectives, and evaluation protocol are kept consistent across different configurations unless otherwise specified. We evaluate memory capability using the same memory benchmarks to characterize performance changes across different scales.

所有扩展实验均基于 MemT-1.0-Nano，遵循受控的实验协议。 我们探索两个扩展方向：数据与训练扩展，以及记忆容量扩展。 除非另有说明，不同配置的骨干架构、训练目标和评估协议均保持一致。 我们使用相同的记忆基准评估记忆能力，以刻画不同规模下的性能变化。

Specifically, for data and training scaling, we evaluate checkpoints along a continuous post-training trajectory spanning two consecutive epochs over the same corpus. The model architecture and training corpus remain fixed, while cumulative training exposure and optimization steps increase throughout training. For memory capacity scaling, we vary the number of memory experts in MoS to expand native memory capacity, while keeping the remaining architectural components and training configurations consistent. By examining the resulting performance curves, we investigate how native memory capability evolves with additional post-training and increased memory capacity.

具体而言，对于数据与训练扩展，我们沿同一语料上连续两个 epoch 的后训练轨迹评估检查点。 模型架构与训练语料保持固定，而累计训练数据接触量和优化步数随训练推进而增长。 对于记忆容量扩展，我们改变 MoS 中的记忆专家数量，以扩大原生记忆容量，同时保持其余架构组件和训练配置一致。 通过考察所得性能曲线，我们研究原生记忆能力如何随额外后训练和容量增加而演变。

5.3. Data and Training Scaling

We explore whether additional exposure to memory-related post-training data improves memory capability at a fixed model capacity. Starting from the CPT checkpoint, we follow a single MemT-1.0-Nano training trajectory for two epochs over the same post-training corpus.

我们探索在模型容量固定时，进一步接触记忆相关后训练数据能否提升记忆能力。 从 CPT 检查点开始，我们跟踪 MemT-1.0-Nano 在同一后训练语料上连续训练两个 epoch 的单条训练轨迹。

图6：MemT-1.0-Nano 的数据与训练扩展。在连续两个后训练 epoch 中的记忆性能。蓝色圆点表示左轴上的平均得分，红色方块表示右轴上的 LoCoMo 准确率。

Figure 6 shows that memory performance generally improves with additional post-training. Across the sampled checkpoints, the average score increases from 36.65 to 46.80, with intermediate fluctuations. Most of this gain arrives early: the average score rises steeply within the first quarter epoch and continues to climb over the remainder of the first epoch, reaching 45.40 by its end, while LoCoMo improves from 40.39 to 55.39 over the same span. This pattern is consistent with rapid adaptation of the CPT-initialized memory modules to downstream query answering, after which the increments shrink steadily but do not disappear.

图6 表明，记忆性能总体上随额外后训练而提高。 在采样的检查点之间，平均得分从 36.65 升至 46.80，期间存在波动。 大部分收益出现在早期：平均得分在前四分之一个 epoch 内迅速上升，并在第一个 epoch 的剩余阶段继续提高，到结束时达到 45.40；同期 LoCoMo 从 40.39 升至 55.39。 这一模式与 CPT 初始化的记忆模块快速适应下游问答相吻合，之后增量持续缩小，但并未消失。

In the second epoch, the final sampled checkpoint reaches 46.80 on average and 57.53 on LoCoMo, an improvement over the last sampled checkpoint of the first epoch despite intermediate fluctuations. These gains indicate remaining room for optimization at fixed memory capacity, even when training revisits previously seen data. In summary, continued exposure to memory-related post-training data can improve native memory capability, with further gains achievable across epochs without increasing model capacity.

在第二个 epoch 中，最后一个采样检查点的平均得分达到 46.80、LoCoMo 达到 57.53，尽管中间存在波动，仍优于第一个 epoch 的最后一个采样检查点。 这些收益表明，即使重新学习已经见过的数据，在记忆容量固定时仍存在优化空间。 综上，持续接触记忆相关后训练数据可以提升原生记忆能力，即使不增加模型容量，也能在后续 epoch 获得进一步收益。

5.4. Memory Capacity Scaling

Following 400 steps of CPT, we explore memory capacity scaling during post-training by comparing MoS configurations with 2 and 4 memory experts. We evaluate both of them at 200, 3,400, and 5,500 post-training steps, reporting the average score alongside the LoCoMo score. The results are presented in Figure 7. The 4-expert configuration consistently outperforms the 2-expert configuration across all three checkpoints, with relatively modest gains of 0.41 points in the average score and 1.42 points on LoCoMo at 200 post-training steps.

经过 400 步 CPT 后，我们在后训练阶段比较具有 2 个和 4 个记忆专家的 MoS 配置，探索记忆容量扩展。 我们在后训练 200、3,400 和 5,500 步时评估这两个配置，同时报告平均得分和 LoCoMo 得分。 结果见 图7。 在三个检查点上，4 专家配置始终优于 2 专家配置；在后训练 200 步时，收益相对较小，平均得分提高 0.41 个点，LoCoMo 提高 1.42 个点。

At 3,400 and 5,500 steps, increasing the number of experts improves the average score by 2.01 and 2.69 points, respectively, and LoCoMo by 3.25 and 2.20 points. The 4-expert configuration continues to improve between these later checkpoints, with its average score increasing from 44.29 to 44.68 and its LoCoMo score rising from 53.51 to 55.58. These results suggest that expanding memory capacity through additional experts benefits downstream performance, with larger gains at the later post-training checkpoints than at 200 steps.

在 3,400 和 5,500 步时，增加专家数量使平均得分分别提高 2.01 和 2.69 个点，LoCoMo 分别提高 3.25 和 2.20 个点。 4 专家配置在这两个较晚检查点之间仍继续改善，平均得分从 44.29 升至 44.68，LoCoMo 从 53.51 升至 55.58。 这些结果表明，通过增加专家来扩展记忆容量有利于下游性能，且在较晚的后训练检查点上，收益大于 200 步时。

图7：经过 400 步 CPT 后，后训练期间的记忆容量扩展。我们在相同后训练步数下比较具有 2 个和 4 个记忆专家的 MoS 配置，报告平均得分和 LoCoMo 得分。

6. Evaluation

6.1. Overall Evaluation Setup

We evaluate MemT-1.0 from the perspectives of memory and general capabilities. The memory evaluation focuses on whether MemT-1.0 can answer questions based on its dynamic memory states, without access to prior textual context. The evaluation on general capabilities examines the impact of the additional native memory mechanism on the model’s general capabilities. These aspects provide an important foundation for self-evolving memory-native models.

我们从记忆能力和通用能力两个角度评估 MemT-1.0。 记忆评估关注 MemT-1.0 能否在无法访问此前文本上下文的情况下，仅根据动态记忆状态回答问题。 通用能力评估则考察额外的原生记忆机制对模型通用能力的影响。 这些方面为能够自我演化的记忆原生模型提供了重要基础。

Benchmarks. We evaluate memory capabilities on 11 benchmarks and divide them into four groups according to trajectory length. Specifically, the group below 16K includes MemBench and ImplicitMemBench, and the 16K-32K group includes PersonaMem, PS-Bench, and LoCoMo. For longer trajectories, the 32K-64K group includes PerMemSafe, Memora, and MemOps, and the group above 64K includes LongMemEval-S, MemoryAgentBench, and BEAM-1M. These benchmarks focus on long-term information retention and retrieval, dynamic memory updating and management, and personalized understanding.

基准。 我们在 11 个基准上评估记忆能力，并按轨迹长度分为四组。 具体来说，低于 16K 的一组包括 MemBench 和 ImplicitMemBench；16K–32K 一组包括 PersonaMem、PS-Bench 和 LoCoMo。 对于更长的轨迹，32K–64K 一组包括 PerMemSafe、Memora 和 MemOps；高于 64K 的一组包括 LongMemEval-S、MemoryAgentBench 和 BEAM-1M。 这些基准关注长期信息保留与检索、动态记忆更新与管理，以及个性化理解。

In addition, we evaluate general capabilities on six benchmarks spanning four major dimensions. Specifically, we evaluate knowledge and scientific reasoning with MMLU-Pro and GPQA Diamond, instruction following with IFEval, long-context understanding with LongBench v2, and code generation with BigCodeBench-Hard and LiveCodeBench v6. These benchmarks collectively assess a wide range of general capabilities, thereby reflecting the impact of the additional native memory mechanism on the underlying backbone model. More details are available in Appendix D.1.

此外，我们在覆盖四大维度的六个基准上评估通用能力。 具体来说，使用 MMLU-Pro 和 GPQA Diamond 评估知识与科学推理，使用 IFEval 评估指令遵循，使用 LongBench v2 评估长上下文理解，并使用 BigCodeBench-Hard 和 LiveCodeBench v6 评估代码生成。 这些基准共同评估广泛的通用能力，从而反映额外原生记忆机制对底层骨干模型的影响。 更多细节见 附录 D.1。

Baselines. We compare MemT-1.0 with textual memory and parametric memory approaches. Textual memory methods perform inference based on the original backbone model with full context and partial context settings. Full context provides the complete historical text as memory in the input context, while DenseRAG retrieves a subset of relevant information as contextual evidence. We also include representative memory systems for comparison, including MemOS, Mem0, OpenViking, and Graphiti.

基线。 我们将 MemT-1.0 与文本记忆方法和参数化记忆方法比较。 文本记忆方法基于原始骨干模型，在完整上下文和部分上下文设置下推理。 完整上下文将全部历史文本作为记忆放入输入上下文，DenseRAG 则检索相关信息子集，作为上下文证据。 我们还纳入具有代表性的记忆系统进行比较，包括 MemOS、Mem0、OpenViking 和 Graphiti。

For parametric memory methods, no textual history is provided in the context, and the model performs inference using its own parametric memory. Our parametric memory baselines include Temp-LoRA, Doc-to-LoRA, SHINE, $\delta$-mem, LiveMem, MemoryLLM and ICAE. More details of the baselines are provided in Appendix D.2.

对于参数化记忆方法，上下文中不提供文本历史，模型使用自身的参数化记忆推理。 参数化记忆基线包括 Temp-LoRA、Doc-to-LoRA、SHINE、$\delta$-mem、LiveMem、MemoryLLM 和 ICAE。 基线的更多细节见 附录 D.2。

Evaluation Protocol. For memory capability evaluation, in each trajectory, we first reset the memory and write the historical information into memory following the chronological order. Then, the model is required to answer multiple questions related to the historical information, and the responses are compared against the ground-truth answers to compute accuracy. Specifically, memory is strictly isolated across different users and histories. Ground-truth answers, evidence annotations, and evaluation labels are only used for scoring and are never exposed to the evaluated models.

评估协议。 对于记忆能力评估，每条轨迹先重置记忆，再按时间顺序将历史信息写入记忆。 随后，要求模型回答多个与历史信息有关的问题，并将回复与真实答案比较，以计算准确率。 具体来说，不同用户和不同历史之间的记忆严格隔离。 真实答案、证据标注和评估标签仅用于评分，绝不提供给被评估模型。

For LLM-as-a-Judge, we utilize gpt-4.1-mini with the OmniMemEval prompt. For general capability evaluation, we introduce irrelevant information before inference, which aims to examine how the native memory mechanism affects the model’s general capabilities under the noise setting. More details are available in Appendix D.4.

对于 LLM-as-a-Judge，我们采用 gpt-4.1-mini，并使用 OmniMemEval 提示词。 对于通用能力评估，我们在推理前引入无关信息，目的是考察噪声设置下原生记忆机制如何影响模型的通用能力。 更多细节见 附录 D.4。

表2：11 个基准上的记忆能力评估。基准按轨迹长度分组。DS 表示 DeepSeek-V4-Flash，QW 表示 Qwen3.6-35B-A3B。加粗和下划线分别表示各记忆类别内的最佳与次佳结果。

Method	Backbone	<16K		16–32K			32–64K			>64K			All
		MB	IM	PM	PSB	LCM	PMS	MMR	MO	LME	MAB	BEAM	AVG
Textual Memory
MemOS	DS	78.4	68.2	67.1	92.9	85.7	45.9	55.5	77.0	83.2	27.8	72.6	68.6
	QW	79.9	61.1	67.6	97.3	83.4	54.7	52.6	82.6	80.8	21.8	71.3	68.5
Mem0	DS	75.8	45.9	67.2	89.4	79.0	66.0	47.5	75.7	72.6	30.3	71.4	65.5
	QW	78.5	60.4	63.2	97.3	80.9	54.9	41.4	79.4	78.9	20.6	70.3	66.0
OpenViking	DS	75.7	41.4	65.4	95.4	79.1	53.2	53.4	77.3	75.0	28.2	71.9	65.1
	QW	76.8	53.0	61.6	95.9	72.9	53.7	51.5	81.7	82.0	23.2	67.3	65.4
Graphiti	DS	49.6	39.4	53.7	89.0	66.7	30.8	26.2	51.1	57.0	19.9	60.4	49.4
	QW	61.9	34.9	49.8	96.5	50.6	24.5	23.0	47.4	38.1	15.5	60.6	45.7
Full Context	DS	83.2	58.0	65.2	97.4	79.9	32.3	51.7	88.7	77.8	36.0	44.7	65.0
	QW	84.3	53.6	65.2	98.7	83.8	44.8	49.8	92.4	76.8	35.9	43.7	66.3
DenseRAG	DS	69.3	55.0	59.4	89.1	58.3	55.5	31.4	68.5	69.4	30.5	43.7	57.3
	QW	67.1	52.2	60.3	97.7	61.1	45.2	31.3	70.6	70.8	28.2	37.1	56.5
Parametric Memory
MemT-1.0-Flash	–	73.5	49.2	61.6	97.5	65.1	33.5	20.5	64.1	37.2	17.4	37.0	50.6
MemT-1.0-Nano	–	71.8	43.1	52.3	95.7	57.5	38.7	23.9	49.9	32.6	15.5	35.8	47.0
Temp-LoRA	–	44.3	32.0	57.4	98.0	25.1	35.7	16.2	52.5	13.4	12.0	32.3	38.1
Doc-to-LoRA	–	57.2	42.4	34.0	70.2	24.5	16.5	15.3	36.0	14.6	3.54	26.0	30.9
SHINE	–	40.7	20.2	39.9	63.3	26.7	4.40	16.7	39.6	13.8	6.15	9.58	25.6
δ-mem	–	37.5	29.9	46.9	90.8	20.3	20.1	17.9	40.9	14.8	7.13	17.0	31.2
LiveMem-RL	–	35.7	23.3	46.0	82.0	24.4	10.5	16.4	39.8	14.6	7.27	27.1	29.7
MemoryLLM	–	22.7	17.8	41.1	37.4	46.6	3.33	16.3	46.8	15.6	6.26	24.0	25.3
ICAE	–	31.9	24.5	17.8	70.4	22.4	2.80	11.4	40.4	11.0	7.35	11.7	22.9

MB：MemBench；IM：ImplicitMemBench；PM：PersonaMem；PSB：PS-Bench；LCM：LoCoMo；PMS：PerMemSafe；MMR：Memora；MO：MemOps；LME：LongMemEval-S；MAB：MemoryAgentBench；BEAM：BEAM-1M。AVG 为 11 个基准的未加权平均。

6.2. Memory Capabilities

The results for memory capabilities are presented in Table 2. MemT-1.0-Flash and MemT-1.0-Nano achieve average scores of 50.6 and 47.0, respectively, ranking first and second among the parametric memory methods, ahead of the strongest parametric baseline by 12.5 and 8.9 points. The results support native memory as a shared representation for diverse memory tasks without providing historical text in the query context. The comparison between MemT-1.0-Flash and MemT-1.0-Nano reveals distinct strengths across tasks, though the aggregate gap between them remains modest and is driven mainly by memory operations.

记忆能力结果见 表2。 MemT-1.0-Flash 和 MemT-1.0-Nano 的平均得分分别为 50.6 和 47.0，在参数化记忆方法中位列第一和第二，比最强参数化基线分别高 12.5 和 8.9 个点。 这些结果支持将原生记忆作为多种记忆任务的共享表示，而无需在查询上下文中提供历史文本。 MemT-1.0-Flash 与 MemT-1.0-Nano 的比较显示，它们在不同任务上各有所长，但总体差距仍较小，且主要由记忆操作任务驱动。

MemT-1.0-Flash improves most clearly on memory operations, personalized understanding, conversational recall, and implicit learning. MemT-1.0-Nano remains stronger on PerMemSafe and Memora, whereas MemT-1.0-Flash scores higher on Persona-grounded PS-Bench. The advantage over parametric memory baselines also extends to long histories, where MemT-1.0 leads this group on LongMemEval-S, MemoryAgentBench, and BEAM-1M. However, the remaining errors on these benchmarks show that processing an extended trajectory does not by itself ensure reliable access to its contents.

MemT-1.0-Flash 在记忆操作、个性化理解、对话回忆和隐式学习上提升最明显。 MemT-1.0-Nano 在 PerMemSafe 和 Memora 上仍更强，而 MemT-1.0-Flash 在基于人物设定的 PS-Bench 上得分更高。 相较参数化记忆基线的优势还延伸到长历史，在 LongMemEval-S、MemoryAgentBench 和 BEAM-1M 上，MemT-1.0 均领先于该组方法。 但这些基准上仍存在的错误说明，仅仅处理一条长轨迹，并不能保证可靠访问其中内容。

Answering later queries requires earlier information to survive successive updates and remain distinguishable from other stored associations. The results motivate further improvements in selective retention and query-dependent readout.

要回答之后的查询，较早信息必须经受连续更新，并与其他已存储关联保持可区分性。 这些结果促使我们进一步改进选择性保留和依赖查询的读出。

The strongest textual memory systems still achieve substantially higher performance, particularly on LongMemEval-S and BEAM-1M. These systems provide explicit historical evidence at query time, whereas MemT-1.0 must recover the required information from its evolving states. This gap is consistent with the difficulty of preserving detailed evidence through recurrent memory updates and highlights memory fidelity as an important target for further development.

最强的文本记忆系统仍取得明显更高的性能，尤其是在 LongMemEval-S 和 BEAM-1M 上。 这些系统在查询时提供显式历史证据，而 MemT-1.0 必须从持续演化的状态中恢复所需信息。 这一差距与在循环记忆更新中保留详细证据的难度相吻合，并表明记忆保真度是后续发展的重要目标。

6.3. General Capabilities

We evaluate general capabilities under the noise setting. In this setting, we inject irrelevant information into memory before the evaluation tasks, which simulates the impact of distracting memories during real-world online interactions. We report the performance gap between MemT-1.0 and its corresponding backbone model under the noise setting. The results are presented in Table 3.

我们在噪声设置下评估通用能力。 在此设置中，评估任务开始前先向记忆注入无关信息，以模拟真实在线交互中干扰记忆的影响。 我们报告噪声设置下 MemT-1.0 与对应骨干模型之间的性能差距。 结果见 表3。

MemT-1.0-Flash performs comparably to its DeepSeek-V4-Flash backbone on average under the same noise setting. The largest gain occurs on BigCodeBench-Hard, with smaller improvements on IFEval and LongBench v2. Performance decreases modestly on MMLU-Pro, GPQA Diamond, and LiveCodeBench v6. These results suggest that MemT-1.0-Flash broadly preserves general capabilities under distracting memories.

在相同噪声设置下，MemT-1.0-Flash 的平均表现与其 DeepSeek-V4-Flash 骨干相当。 最大收益出现在 BigCodeBench-Hard，在 IFEval 和 LongBench v2 上也有较小提升。 MMLU-Pro、GPQA Diamond 和 LiveCodeBench v6 的表现略有下降。 这些结果表明，MemT-1.0-Flash 在干扰记忆下总体保留了通用能力。

We next examine MemT-1.0-Nano, whose average decrease is around one point. Under the noise setting, most benchmarks remain close to the backbone, with gains on IFEval and BigCodeBench-Hard. The degradation is concentrated rather than uniform. The largest decrease occurs on LiveCodeBench v6, followed by smaller decreases on LongBench v2, MMLU-Pro, and GPQA Diamond. The contrasting changes on BigCodeBench-Hard and LiveCodeBench v6 further suggest that the impact varies even across benchmarks within the same capability domain.

接着考察 MemT-1.0-Nano，其平均下降约为一个点。 在噪声设置下，多数基准的表现仍接近骨干，并在 IFEval 和 BigCodeBench-Hard 上获得提升。 退化集中于特定任务，而非均匀出现。 最大降幅出现在 LiveCodeBench v6，其次是 LongBench v2、MMLU-Pro 和 GPQA Diamond 上较小的降幅。 BigCodeBench-Hard 和 LiveCodeBench v6 上相反的变化进一步表明，即使在同一能力领域的不同基准之间，影响也有所差异。

Overall, the results support broad capability preservation alongside native memory, while highlighting that resistance to distracting memories should be evaluated at the task level.

总体而言，结果表明模型在获得原生记忆的同时，能够广泛保留原有能力，但也强调对干扰记忆的抵抗力应在任务层面评估。

表3：骨干与噪声设置下的通用能力评估。MemT-1.0 的结果报告相对于相同设置下对应骨干的得分差。

Model	MP	GPQA	IFE	LB	BCB-H	LCB-v6	AVG
Backbone
DeepSeek-V4-Flash	85.7	91.2	93.5	63.0	35.8	87.5	76.1
Qwen3.6-35B-A3B	84.6	83.9	90.6	57.3	27.0	88.0	71.9
Noise Setting
DeepSeek-V4-Flash	83.8	89.6	93.0	64.2	32.4	89.2	75.4
MemT-1.0-Flash	−1.8	−0.6	+0.9	+0.2	+2.0	−0.7	0.0
Qwen3.6-35B-A3B	84.4	80.1	89.3	55.7	25.7	88.1	70.5
MemT-1.0-Nano	−1.6	−1.5	+3.3	−2.0	+2.0	−5.8	−0.9

MP：MMLU-Pro；GPQA：GPQA Diamond；IFE：IFEval；LB：LongBench v2；BCB-H：BigCodeBench-Hard；LCB-v6：LiveCodeBench v6。AVG 为六个基准的未加权平均。MemT-1.0 数值表示相对噪声设置下对应骨干的得分差，绿色表示下降，红色表示上升。

6.4. Ablation Studies

In this section, we further present several ablation studies based on MemT-1.0-Nano during model development. As for the model architecture, we focus on the update rule and update granularity in the native memory procedure. We also design a bidirectional encoding mechanism to aggregate information from the current turn. In addition, we study the contribution of different training stages. These experiments further verify the effectiveness of our current architecture.

本节进一步给出模型开发过程中基于 MemT-1.0-Nano 开展的若干消融实验。 对于模型架构，我们关注原生记忆过程中的更新规则和更新粒度。 我们还设计了双向编码机制，用于聚合当前轮次的信息。 此外，研究不同训练阶段的贡献。 这些实验进一步验证了当前架构的有效性。

6.4.1. Architecture Ablation

For architecture ablation, we evaluate four variants. First, we remove the stable mechanism to recover the GDN-2 memory update (w/ GDN-2) by directly using

对于架构消融，我们评估四个变体。 首先，移除稳定机制，直接使用下式恢复 GDN-2 记忆更新（w/ GDN-2）：

$$\begin{array}{c}{\tilde{\mathbf{M}}}_{t,i,h}^{(l)}=\\ [\mathbf{I}-{\mathbf{K}}_{t,i,h}^{(l)\mathrm{\top }}({\mathit{\beta }}_{t,i,h}^{(l)}\odot {\mathbf{K}}_{t,i,h}^{(l)})]\cdot {\mathbf{D}}_{t,i,h}^{(l)}{\tilde{\mathbf{M}}}_{t,i-1,h}^{(l)}\\ +{\mathbf{K}}_{t,i,h}^{(l)\mathrm{\top }}({\mathit{\omega }}_{t,i,h}^{(l)}\odot {\mathbf{V}}_{t,i,h}^{(l)}).\end{array}$$

We also change the update rule into GDN as another ablation baseline (w/ GDN). Second, we change the update granularity from token-level to turn-level (w/ TL). The storage procedure can be defined as

我们也将更新规则改为 GDN，作为另一消融基线（w/ GDN）。 其次，将更新粒度从 token 级改为轮次级（w/ TL）。 存储过程可定义为

$$\begin{array}{c}{\mathbf{M}}_{t+1,h}^{(l)}=\\ [\mathbf{I}-\frac{1}{L_t}{\mathbf{U}}_{t,h}^{(l)\mathrm{\top }}{\mathbf{U}}_{t,h}^{(l)}]\cdot {\mathbf{D}}_{t,h}^{(l)}{\mathbf{M}}_{t,h}^{(l)}\\ +\frac{1}{L_t}{\mathbf{U}}_{t,h}^{(l)\mathrm{\top }}({\mathit{\omega }}_{t,h}^{(l)}\odot {\mathbf{V}}_{t,h}^{(l)}),\end{array}$$

where we have

其中

$${\mathbf{D}}_{t,h}^{(l)}=\mathrm{Diag}[\exp \left(\frac{1}{L_t}\sum _{i=1}^{L_t}{\mathit{\lambda }}_{t,i,h}^{(l)}\right)].$$

We also evaluate a bidirectional encoding (w/ BE) for information aggregation. Specifically, during the storage procedure in each turn, it first projects the pre-attention hidden states ${\tilde{\mathbf{H}}}_t^{(l)}$ into a lower-dimensional space, and applies multi-head self-attention with rotary position embeddings without a causal mask. After that, it projects the result back to the backbone dimension through a residual connection. The enriched representations are used to predict the erase, write, and forget gates, while the memory keys and values are still projected from the original pre-attention hidden states.

我们还评估用于信息聚合的双向编码（w/ BE）。 具体来说，每轮存储时，先将注意力前置隐藏状态 ${\tilde{\mathbf{H}}}_t^{(l)}$ 投影到低维空间，再应用带旋转位置嵌入、无因果掩码的多头自注意力。 之后，通过残差连接将结果投影回骨干维度。 增强后的表示用于预测擦除门、写入门和遗忘门，而记忆键和值仍从原始注意力前置隐藏状态投影得到。

As shown in Table 4, the update granularity has the largest effect among the architecture variants. Replacing token-level with turn-level aggregation reduces the average from 46.97 to 40.98, a far larger drop than any change to the update rule. This may be because successive token-level updates can better account for memory decay and revise associations established earlier within the same turn. The update-rule variants, by contrast, stay close to the full model, with removing the stable mechanism giving 46.43 and substituting GDN giving 46.72.

如 表4 所示，在架构变体中，更新粒度的影响最大。 将 token 级聚合替换为轮次级聚合，使平均得分从 46.97 降至 40.98，降幅远大于对更新规则的任何更改。 这可能是因为连续的 token 级更新能够更好地考虑记忆衰减，并修订同一轮内较早建立的关联。 相比之下，更新规则变体接近完整模型，移除稳定机制得到 46.43，替换为 GDN 得到 46.72。

Both fall within a narrow band around the baseline, so the aggregate capability is not determined by the precise form of the update rule. Adding bidirectional encoding reduces the average to 45.43. Its effect is benchmark-dependent rather than uniformly negative: MB, IM, PM, and PMS improve, whereas the remaining benchmarks regress. Under the evaluated training budget, bidirectional encoding does not improve aggregate memory performance, and its benchmark-specific gains do not offset the losses elsewhere.

二者均处于基线附近的狭窄范围内，因此总体能力并不由更新规则的精确形式决定。 加入双向编码使平均得分降至 45.43。 其影响取决于基准，而非统一为负：MB、IM、PM 和 PMS 提升，其余基准下降。 在所评估的训练预算下，双向编码未改善总体记忆性能，其在特定基准上的收益不足以抵消其他基准的损失。

6.4.2. Training Ablation

We further conduct ablation studies on our training procedures. For the w/o CPT setting, we skip CPT and directly train the memory parameters through post-training alone. For the w/o PT setting, we evaluate the checkpoint after continued pre-training without subsequent post-training. From the results in Table 4, we find that post-training plays an important role in memory-specific tasks, where it increases the average score by approximately 10.69 points over the CPT checkpoint.

我们进一步对训练过程开展消融研究。 对于 w/o CPT 设置，跳过 CPT，仅通过后训练直接训练记忆参数。 对于 w/o PT 设置，评估完成持续预训练、尚未进行后训练的检查点。 从 表4 可见，后训练对记忆专属任务作用显著，相较 CPT 检查点使平均得分提高约 10.69 个点。

These results indicate that language-modeling CPT alone does not provide the same memory-conditioned response capability as the complete pipeline. In addition, the CPT stage improves post-training as well. Therefore, post-training accounts for the larger observed gain in this comparison, while CPT provides an additional aggregate benefit as an initialization stage.

这些结果表明，仅进行语言建模 CPT，无法获得与完整流程相同的记忆条件回复能力。 此外，CPT 阶段也能改善后训练效果。 因此，在这一比较中，后训练贡献了更大的观测收益，而 CPT 作为初始化阶段，带来了额外的总体收益。

表4：MemT-1.0-Nano 的架构与训练消融。加粗和下划线分别表示最佳与次佳结果。Δ AVG 表示相对于 MemT-1.0-Nano 的得分差。

Variant	<16K		16–32K			32–64K			>64K			All
	MB	IM	PM	PSB	LCM	PMS	MMR	MO	LME	MAB	BEAM	AVG	Δ AVG
MemT-1.0-Nano	71.8	43.1	52.3	95.7	57.5	38.7	23.9	49.9	32.6	15.5	35.8	47.0	0.0
Architecture Ablation
w/ GDN-2	71.6	43.7	55.7	96.2	57.3	36.9	21.7	49.2	29.4	14.4	34.6	46.4	-0.5
w/ GDN	72.3	41.9	54.5	98.3	57.5	40.9	20.3	50.6	30.0	13.4	34.2	46.7	-0.3
w/ BE	73.9	44.7	54.3	94.9	55.4	40.8	16.3	45.5	25.6	14.4	34.0	45.4	-1.5
w/ TL	69.1	39.5	52.1	97.5	52.9	36.4	17.2	38.3	25.0	5.5	17.4	41.0	-6.0
Training Ablation
w/o CPT	69.8	37.4	54.7	96.8	55.5	35.3	18.7	50.2	27.0	13.4	34.7	44.9	-2.1
w/o PT	66.2	38.7	41.3	90.2	40.4	26.3	16.1	24.6	13.8	12.2	29.4	36.3	-10.7

6.5. End-to-End Inference Efficiency

We further evaluate the end-to-end inference efficiency of MemT-1.0 against baseline methods, with a particular focus on the results as the history length increases. Specifically, we compare MemT-1.0-Nano with full context, $\delta$-mem, and Temp-LoRA on BEAM. Full context, Temp-LoRA and MemT-1.0-Nano use HF inference on two GPUs exclusively allocated to each run, with backbone layers distributed across devices, a concurrency of 1, and a batch size of 1. For $\delta$-mem, we use its publicly released model and implementation.

我们进一步对照基线方法评估 MemT-1.0 的端到端推理效率，重点考察历史长度增加时的结果。 具体来说，在 BEAM 上比较 MemT-1.0-Nano、完整上下文、$\delta$-mem 和 Temp-LoRA。 完整上下文、Temp-LoRA 和 MemT-1.0-Nano 均采用 HF 推理，每次运行独占两块 GPU，骨干层分布在不同设备上，并发数与批大小均为 1。 对于 $\delta$-mem，我们使用其公开发布的模型与实现。

图8：左：端到端延迟包括历史准备和生成 128 token 回复。右：查询 TTFT 测量从查询开始执行到生成第一个 token 的时间，不包括历史准备。

We construct input sequences with varying history lengths based on BEAM. The query uses shared 512-token content, with the chat template added separately, rather than a benchmark question, and each sequence is treated independently. We evaluate a fixed generation length of 128 tokens by forcing the model to decode exactly 128 tokens while ignoring the end-of-sequence signal. For the HF evaluations, synchronized wall-clock timers exclude model loading, input tokenization, and evaluation-related scoring overhead.

我们基于 BEAM 构建历史长度不同的输入序列。 查询采用共享的 512 token 内容，另行添加聊天模板，而非使用基准问题；各序列独立处理。 我们忽略序列结束信号，强制模型恰好解码 128 个 token，以评估固定为 128 token 的生成长度。 对于 HF 评估，同步的实际耗时计时器排除模型加载、输入分词和评估相关评分开销。

For each history and length, we measure preparation once, discard one query warm-up, and average five timed queries. Reported curves average the resulting values across 35 histories. For MemT-1.0-Nano, history content is processed in 2,048-token slices, with each chat-formatted chunk capped at 2,048 tokens.

对于每个历史及长度，测量一次准备过程，丢弃一次查询预热，再对五次计时查询取平均。 报告的曲线进一步在 35 个历史上，对上述结果取平均。 对于 MemT-1.0-Nano，历史内容按 2,048 token 切片处理，每个加入聊天格式后的分块上限为 2,048 token。

The reported end-to-end latency includes history preparation and response generation, accounting for indexing, test-time adaptation, or memory writes as applicable. We additionally report query time to first token (TTFT), measured from the start of query execution to the first generated token, including query prefilling but excluding history preparation. For full context and MemT-1.0-Nano, both end-to-end latency and TTFT include state restoration. The HF full-context baseline reuses the prepared history cache.

报告的端到端延迟包括历史准备与回复生成，并视方法计入索引、测试时适应或记忆写入。 此外，我们报告查询首 token 时间（TTFT），从查询开始执行计时到生成第一个 token，包括查询预填充，但不包括历史准备。 对于完整上下文和 MemT-1.0-Nano，端到端延迟与 TTFT 都包括状态恢复。 HF 完整上下文基线复用已准备好的历史缓存。

We report results over the common history-length range from 512 to 256K tokens. Figure 8 presents the results. MemT-1.0-Nano achieves an end-to-end advantage over full context at long histories. It is generally slower at short histories and approximately matches full context at 32K tokens, reaching speedups of approximately $1.20\times$ at 64K, $1.86\times$ at 128K, and $3.99\times$ at 256K.

我们在共同覆盖的 512 到 256K token 历史长度范围内报告结果。 结果见 图8。 在长历史下，MemT-1.0-Nano 相较完整上下文取得端到端优势。 短历史下它通常更慢，在 32K token 时与完整上下文大致持平；64K、128K 和 256K 时，加速比分别约为 $1.20\times$、$1.86\times$ 和 $3.99\times$。

Its memory storage procedure processes history incrementally; generation then accesses the resulting state without retaining historical tokens in the attention context. In comparison, the full-context baseline incurs both history prefilling and continued attention over historical KV entries during generation. The measured long-history end-to-end advantage primarily reflects lower history-preparation cost.

它的记忆存储过程以增量方式处理历史，之后生成访问得到的状态，而不在注意力上下文中保留历史 token。 相比之下，完整上下文基线既需要历史预填充，又在生成期间持续对历史 KV 条目进行注意力计算。 测得的长历史端到端优势主要反映了更低的历史准备成本。

The query TTFT curves reveal an additional benefit after history preparation. MemT-1.0-Nano maintains broadly stable mean query TTFT of approximately 104–112 ms across the evaluated range, while full-context TTFT rises to approximately 257 ms at 128K tokens and 445 ms at 256K tokens. MemT-1.0-Nano's behavior is consistent with a fixed-size memory readout whose state dimensions do not grow with the number of historical tokens processed. For persistent interactions, it allows accumulated history to influence responses without progressively increasing the attention context used for each new query. The benefit concerns access to the stored representation, and the memory-capability results above already show that this representation preserves useful information. At 128K and 256K tokens, MemT-1.0-Nano achieves query TTFT speedups of approximately $2.44\times$ and $4.19\times$, respectively, over cached full-context inference.

查询 TTFT 曲线揭示了历史准备完成后的额外收益。 在所评估范围内，MemT-1.0-Nano 的平均查询 TTFT 基本稳定在约 104–112 毫秒，而完整上下文的 TTFT 在 128K token 时升至约 257 毫秒，在 256K token 时升至约 445 毫秒。 MemT-1.0-Nano 的表现与固定大小的记忆读出一致，其状态维度不会随已处理历史 token 的数量增长。 对于持久交互，这使累积历史能够影响回复，而不必逐步扩大每个新查询使用的注意力上下文。 这一收益涉及对已存储表示的访问；上面的记忆能力结果已经表明，该表示保留了有用信息。 在 128K 和 256K token 时，相较缓存的完整上下文推理，MemT-1.0-Nano 的查询 TTFT 加速比分别约为 $2.44\times$ 和 $4.19\times$。

Previously collected results for the parametric memory baselines highlight the importance of accounting for preparation costs. Temp-LoRA maintains stable query TTFT, but its end-to-end cost grows substantially because history-specific adaptation requires optimization. Although $\delta$-mem has the lowest query TTFT, its end-to-end latency exceeds Nano's from 64K tokens onward as history-preparation cost grows. Absolute latency differences across these systems also reflect their backbone, runtime, and tokenization configurations. These results show the value of efficient memory construction together with predictable query TTFT. Reusing a prepared state amortizes its construction cost over multiple queries.

此前收集的参数化记忆基线结果突出了计入准备成本的重要性。 Temp-LoRA 的查询 TTFT 保持稳定，但由于针对具体历史的适应需要优化，其端到端成本显著增长。 尽管 $\delta$-mem 的查询 TTFT 最低，随着历史准备成本增长，其端到端延迟从 64K token 起便超过 Nano。 这些系统之间绝对延迟的差异，也反映了骨干、运行时和分词配置的不同。 这些结果表明，高效记忆构建与可预测查询 TTFT 的结合具有价值。 复用准备好的状态，可以将其构建成本分摊到多个查询中。

6.6. Storage Overhead

We evaluate the per-trajectory storage overhead on BEAM histories, after history preparation and before querying. The reported storage size is the logical size in bytes of the per-trajectory state. The shared backbone weights, embedders and hyper memory parameters are excluded from the measurement. More details can be found in Appendix D.5, and the results are presented in Figure 9.

我们在 BEAM 历史上，评估每条轨迹在历史准备完成后、查询开始前的存储开销。 报告的存储大小是每条轨迹状态的逻辑字节数。 测量不包括共享骨干权重、嵌入模型和超记忆参数。 更多细节见 附录 D.5，结果见 图9。

Figure 9 shows a clear difference between retaining historical records and maintaining a fixed-size state. Full context incurs increasing storage as its logical state size grows with the visible history. DenseRAG has a much smaller storage overhead at short histories, but its stored texts and retrieval vectors also accumulate as more information is added. In contrast, MemT-1.0-Nano's memory state storage remains constant throughout the trajectory, because new information updates memory states without enlarging them. At a history length of 128K tokens, MemT-1.0-Nano achieves an approximately $66\times$ reduction in per-trajectory storage compared with the full context setting. Temp-LoRA also exhibits constant storage, since its adapter dimensions are fixed, although its storage overhead is larger than MemT-1.0-Nano's in the evaluated configuration.

图9 展示了保留历史记录与维护固定大小状态之间的明显差异。 完整上下文的逻辑状态大小随可见历史增长，因而存储需求不断增加。 DenseRAG 在短历史下的存储开销小得多，但随着信息增加，其存储文本和检索向量也不断累积。 相比之下，MemT-1.0-Nano 的记忆状态存储量在整条轨迹中保持恒定，因为新信息只更新记忆状态，不扩大状态规模。 在历史长度为 128K token 时，相较完整上下文设置，MemT-1.0-Nano 的每条轨迹存储量约缩减为原来的 $1/66$。 Temp-LoRA 的适配器维度固定，因此存储量也恒定，但在所评估配置中，其存储开销高于 MemT-1.0-Nano。

The comparison with DenseRAG illustrates how the storage advantage depends on interaction length. For short histories, storing a small collection of text chunks and embeddings is cheaper than allocating MemT-1.0-Nano's full memory state. As histories grow, DenseRAG's accumulated records eventually exceed that fixed allocation. MemT-1.0-Nano therefore becomes more attractive for persistent sessions with extended interaction histories, where additional turns do not increase the size of the saved state.

与 DenseRAG 的比较说明了存储优势如何依赖交互长度。 对于短历史，存储少量文本块与嵌入，比为 MemT-1.0-Nano 分配完整记忆状态更便宜。 随着历史增长，DenseRAG 累积的记录最终会超过这一固定分配量。 因此，对于交互历史较长的持久会话，MemT-1.0-Nano 更有吸引力，因为新增轮次不会增加已保存状态的大小。

The full context curve ends at its context-window limit, so its reported storage does not represent retaining the entire longest history. For deployment of MemT-1.0, a constant state size makes per-trajectory logical-state budgets predictable, and bounds the logical payload that must be saved or transferred when resuming an interaction. Total storage still grows with the number of independent trajectories and the configured memory capacity.

完整上下文曲线止于其上下文窗口上限，因此报告的存储量并不代表保留了最长历史的全部内容。 对于 MemT-1.0 的部署，恒定状态大小使每条轨迹的逻辑状态预算可预测，并约束了恢复交互时必须保存或传输的逻辑负载大小。 总存储量仍随独立轨迹数量和所配置记忆容量增长。

图9：存储开销结果。历史长度不断增加时，BEAM 历史上每条轨迹的持久存储量。

6.7. Low-Rank Decomposition

To further explore the compressibility of memory states based on MemT-1.0-Nano, we perform online low-rank projection for memory states to study their performance impacts. Specifically, we apply singular value decomposition (SVD) to the FP32 state matrix of each layer after each storage procedure, and reconstruct a dense state from the retained components for subsequent computation. For a retained rank $k$, the reconstructed state is computed as

为进一步探索 MemT-1.0-Nano 记忆状态的可压缩性，我们对记忆状态进行在线低秩投影，研究其性能影响。 具体来说，每次存储过程之后，对每层的 FP32 状态矩阵应用奇异值分解（SVD），并根据保留分量重构稠密状态，用于后续计算。 对于保留秩 $k$，重构状态计算如下：

$${\mathbf{M}}_{t,h}^{(l)}=U_{:,1:k}{\mathrm{\Sigma }}_{1:k,1:k}{V}_{:,1:k}^{\mathrm{\top }}.$$

We set the rank $k\in \{1,2,4,8,16,32,64,128,256\}$, with the full rank $k=256$ retaining the original dense state without projection. The evaluation is conducted on PersonaMem and LoCoMo. For each retention ratio, we independently initialize the memory state, write the complete history, and apply low-rank reconstruction after every memory storage when $k<256$.

我们设置秩 $k\in \{1,2,4,8,16,32,64,128,256\}$，其中满秩 $k=256$ 保留原始稠密状态，不进行投影。 评估在 PersonaMem 和 LoCoMo 上进行。 对每个保留比例，都独立初始化记忆状态，写入完整历史，并在 $k<256$ 时于每次记忆存储后进行低秩重构。

Figure 10 shows different results across the two tasks. On LoCoMo, accuracy generally improves as more singular components are retained. The largest gains fall in the middle of the rank range, and the final doublings add progressively less, although the score still improves up to the full rank. The one exception is the smallest retained rank, which scores slightly above the next one.

图10 显示两个任务上的结果有所不同。 在 LoCoMo 上，保留更多奇异分量时，准确率总体上升。 最大收益出现在秩范围的中部，最后几次翻倍带来的增益逐步减小，尽管得分仍一直提升到满秩。 唯一例外是最小保留秩，其得分略高于下一个秩。

At this level of truncation, the small score difference alone does not establish an advantage of extreme compression. PersonaMem follows the same shape but saturates much earlier. Its accuracy rises quickly and then flattens out before reaching full rank, with the highest observed accuracy at an intermediate rank and no further improvement once additional components are retained. Overall, memory performance improves as the retained rank increases, but the returns diminish quickly, and on PersonaMem they eventually vanish.

在如此强的截断下，仅凭这一微小得分差异，不能确立极端压缩的优势。 PersonaMem 呈现相同形态，但更早饱和。 其准确率迅速上升，随后在达到满秩之前便趋于平缓，最高观测准确率出现在中间秩，继续保留更多分量不再带来提升。 总体而言，记忆性能随保留秩增大而改善，但收益迅速递减，在 PersonaMem 上最终消失。

Because theoretical factor storage grows roughly linearly with the retained rank, increasing the rank yields diminishing returns in downstream accuracy.

由于理论上的分解因子存储量大致随保留秩线性增长，提高秩带来的下游准确率收益逐渐减小。

图10：在线低秩分解对记忆性能和存储的影响。

7. Application in Agentic Tasks

7.1. Overview

In this section, we investigate the potential of MemT-1.0 for developing self-evolving agents through continual learning. We apply MemT-1.0 to long-horizon agentic tasks, where agents must use environment feedback to guide task execution. Real-world agent interactions generate a wealth of trajectories containing valuable procedural experience. Reusing this experience offers a promising path toward improving agent performance across tasks. Current methods typically extract experience from historical trajectories and include it in the textual context of subsequent tasks, increasing the burden on an already growing execution context. MemT-1.0 can instead store experience in native memory and use it to guide actions in long-horizon tasks.

本节考察 MemT-1.0 通过持续学习支持自我演化智能体的潜力。 我们将 MemT-1.0 应用于长程智能体任务，其中智能体必须利用环境反馈指导任务执行。 真实智能体交互会产生大量轨迹，蕴含有价值的程序性经验。 复用这些经验，是提高智能体跨任务表现的一条有前景的路径。 现有方法通常从历史轨迹中提取经验，再将其放入后续任务的文本上下文，加重本已不断增长的执行上下文负担。 MemT-1.0 则可以将经验存入原生记忆，并用其指导长程任务中的动作。

We focus on agentic coding and train MemT-1.0-Nano to apply procedural experience extracted from historical trajectories. We evaluate it on SWE-bench Verified and SWE-bench Multilingual against Qwen3.6-35B-A3B, both without experience and with textual experience. With experience stored before execution, MemT-1.0-Nano improves the proportion of tasks solved in all three runs over the backbone without experience by 7.6 percentage points on SWE-bench Verified and 8.3 points on SWE-bench Multilingual. Results show that MemT-1.0 can improve agentic task performance beyond memory-focused evaluations and support its potential as a foundation for self-evolving agents.

我们聚焦智能体编程，训练 MemT-1.0-Nano 运用从历史轨迹中提取的程序性经验。 我们在 SWE-bench Verified 和 SWE-bench Multilingual 上，将其与不使用经验及使用文本经验的 Qwen3.6-35B-A3B 比较。 当经验在执行前存储时，相较无经验骨干，MemT-1.0-Nano 将三次运行全部成功的任务比例，在 SWE-bench Verified 上提高 7.6 个百分点，在 SWE-bench Multilingual 上提高 8.3 个百分点。 结果表明，MemT-1.0 能够在以记忆为核心的评估之外提升智能体任务性能，支持其作为自我演化智能体基础的潜力。

7.2. Experience-Guided Agentic Framework

To explore this potential, we develop an agentic framework in which MemT-1.0 stores and reuses procedural experience during long-horizon task execution. We construct an experience bank by summarizing historical agent trajectories into reusable procedural guidance. As illustrated in Figure 11, the framework retrieves relevant experience from the bank and stores it in either the agent's textual context or MemT-1.0-Nano's native memory. The two approaches use the same experience but differ in how it is accessed during task execution.

为探索这一潜力，我们开发了一个智能体框架，使 MemT-1.0 在长程任务执行中存储并复用程序性经验。 我们将历史智能体轨迹总结为可复用的程序性指导，构建经验库。 如 图11 所示，框架从经验库中检索相关经验，并将其存入智能体的文本上下文，或 MemT-1.0-Nano 的原生记忆。 两种方式使用相同经验，但在任务执行期间访问经验的方式不同。

Experience Retrieval and Storage. Given a coding task, we retrieve relevant experiences from the experience bank using BGE-M3. Each experience describes its applicable conditions, recommended procedures, and verification strategies. The task description serves as the retrieval query, and candidate experiences are ranked according to embedding similarity. The selected experience is converted into a structured textual representation before storage.

经验检索与存储。 给定编程任务，我们使用 BGE-M3 从经验库检索相关经验。 每条经验描述其适用条件、推荐过程和验证策略。 任务描述作为检索查询，候选经验按嵌入相似度排序。 选中的经验在存储之前被转换为结构化文本表示。

We consider two experience storage settings. In the Before Execution setting, experience is stored before the agent begins execution. In the Harness-Triggered setting, the agent first executes the task independently, and experience is stored when the execution harness detects a critical breakpoint. Such breakpoints are identified using execution feedback, including command failures, empty observations, and repeated unsuccessful operations. The detection only relies on information available during execution, without accessing reference solutions. Each trajectory receives at most one experience store operation.

我们考虑两种经验存储设置。 在 Before Execution 设置中，智能体开始执行之前先存储经验。 在 Harness-Triggered 设置中，智能体先独立执行任务，当执行框架检测到关键断点时再存储经验。 这些断点根据执行反馈识别，包括命令失败、空观测和反复失败的操作。 检测仅依赖执行期间可用的信息，不访问参考解答。 每条轨迹至多接收一次经验存储操作。

Memory-Based Experience Application. In the Textual Experience setting, the retrieved experience is directly inserted into the agent's message history and remains available through the subsequent context. In the Native Memory setting, MemT-1.0-Nano writes the same experience into its native memory state through a separate memory store operation. Specifically, the experience text is processed by a forward pass with memory writing enabled, updating the memory state $M$ without modifying the original task context. After the memory store operation, agent actions access the stored experience through memory attention, allowing the experience to influence action generation without explicitly appearing in the message history.

基于记忆的经验应用。 在 Textual Experience 设置中，检索得到的经验直接插入智能体的消息历史，在后续上下文中持续可用。 在 Native Memory 设置中，MemT-1.0-Nano 通过独立的记忆存储操作，将相同经验写入原生记忆状态。 具体来说，通过启用记忆写入的一次前向传播处理经验文本，更新记忆状态 $M$，而不修改原始任务上下文。 存储操作之后，智能体通过记忆注意力访问已存储经验，使经验能够影响动作生成，而无需显式出现在消息历史中。

Each execution trajectory maintains an independent memory state. The state is initialized to empty before the task and updated when experience is stored. After storing experience, it remains fixed and is read during subsequent execution until it is reset at the end of the trajectory. This setup evaluates whether a trained native-memory agent can effectively reuse procedural experience without retaining the experience text in the task context.

每条执行轨迹维护独立的记忆状态。 该状态在任务开始前初始化为空，并在存储经验时更新。 经验存储后，状态保持固定，并在后续执行中被读取，直到轨迹结束时重置。 这一设置评估训练好的原生记忆智能体能否有效复用程序性经验，而不在任务上下文中保留经验文本。

图11：基于 MemT-1.0-Nano 的经验引导智能体框架。检索得到的经验被存入文本上下文或原生记忆，以指导后续动作。

7.3. Experience Construction and Model Training

To enable experience-guided agentic execution through native memory, we construct training data that associate reusable procedural experience with successful execution behaviors. The overall data construction consists of two stages, extracting reusable experience from historical trajectories and generating experience-conditioned trajectories.

为实现通过原生记忆进行经验引导的智能体执行，我们构建将可复用程序性经验与成功执行行为关联起来的训练数据。 整体数据构建包括两个阶段：从历史轨迹提取可复用经验，以及生成以经验为条件的轨迹。

Experience Data Construction. We first construct an experience bank from 41,057 successful coding trajectories released by Scale-SWE. Following the experience summarization framework of MSCE, the construction process consists of trajectory annotation, procedural policy induction, policy clustering, and experience generation with quality auditing. For each successful trajectory, we identify important execution decisions and their corresponding environmental conditions. Trajectories with similar conditions are grouped to induce reusable procedural policies, which are subsequently consolidated and converted into structured experiences. Each experience contains information about its triggering conditions, recommended execution steps, applicability constraints, and verification procedures. After quality filtering, we obtain an experience bank containing 1,040 reusable coding experiences.

经验数据构建。 首先，从 Scale-SWE 发布的 41,057 条成功编程轨迹中构建经验库。 沿用 MSCE 的经验总结框架，构建过程包括轨迹标注、程序性策略归纳、策略聚类，以及带质量审核的经验生成。 对于每条成功轨迹，我们识别重要执行决策及其对应的环境条件。 具有相似条件的轨迹被分组，用于归纳可复用的程序性策略，随后进行整合，并转换为结构化经验。 每条经验包含触发条件、推荐执行步骤、适用约束和验证过程等信息。 质量过滤后，得到包含 1,040 条可复用编程经验的经验库。

Training Sample Construction. We further construct experience-guided training trajectories through counterfactual execution. Specifically, we generate original coding trajectories using Qwen3.6-35B-A3B with the AweAgent harness in the Scale-SWE environment. For unsuccessful trajectories, we identify candidate breakpoints according to execution errors and other observable failure signals. At each selected breakpoint, we retrieve candidate experiences and resume execution with different experience combinations. The resulting trajectories are evaluated using executable tests, and we retain experience-conditioned continuations that pass the task tests when the corresponding original trajectory failed.

训练样本构建。 我们进一步通过反事实执行，构建经验引导的训练轨迹。 具体来说，在 Scale-SWE 环境中，使用 Qwen3.6-35B-A3B 和 AweAgent 执行框架生成原始编程轨迹。 对于未成功的轨迹，根据执行错误和其他可观测失败信号识别候选断点。 在每个选定断点检索候选经验，并使用不同经验组合恢复执行。 用可执行测试评估所得轨迹；当对应原始轨迹失败时，保留那些在经验条件下通过任务测试的续接轨迹。

Each training sample is represented as

每个训练样本表示为

$$x_{b,j}=({\xi }_{\le b},g_{b,j},{\hat{\xi }}_{>b}^{+}),$$

where $b$ indexes the selected breakpoint and $j$ indexes the candidate experience subset. Here, ${\xi }_{\le b}$ denotes the original trajectory prefix before and including breakpoint $b$, $g_{b,j}$ denotes the injected experience subset, and ${\hat{\xi }}_{>b}^{+}$ denotes the successful continuation after experience injection.

其中，$b$ 索引选定断点，$j$ 索引候选经验子集。 这里，${\xi }_{\le b}$ 表示原始轨迹到断点 $b$ 为止、包含该断点的前缀，$g_{b,j}$ 表示注入的经验子集，${\hat{\xi }}_{>b}^{+}$ 表示注入经验之后的成功续接轨迹。

The retained samples satisfy the condition that the original trajectory fails while the experience-conditioned continuation succeeds. These continuations provide supervision for predicting successful actions conditioned on procedural experience. We generate 18,820 original trajectories and evaluate 199,500 breakpoint-experience intervention variants. Among these interventions, 45,481 produce successful continuations.

保留的样本满足：原始轨迹失败，而以经验为条件的续接轨迹成功。 这些续接轨迹为根据程序性经验预测成功动作提供监督。 我们生成了 18,820 条原始轨迹，并评估 199,500 个断点—经验干预变体。 其中，45,481 次干预产生成功续接轨迹。

After additional quality filtering, we obtain 42,403 training samples. The resulting samples are partitioned by task to avoid leakage between training and validation trajectories. Appendix E provides details of the experience bank, data construction pipeline, and filtering procedure.

经过进一步质量过滤，得到 42,403 个训练样本。 所得样本按任务划分，以避免训练和验证轨迹之间发生泄漏。 附录 E 提供经验库、数据构建流程和过滤过程的详细信息。

Training Strategy. We initialize the model from MemT-1.0-Nano. During application-specific post-training, all backbone parameters are frozen, and only the native hyper memory blocks are optimized. Each training sample is processed through separate experience and main-trajectory streams. First, the experience is written into native memory through a memory store operation.

训练策略。 我们从 MemT-1.0-Nano 初始化模型。 在面向应用的后训练期间，冻结全部骨干参数，仅优化原生超记忆块。 每个训练样本通过相互独立的经验流和主轨迹流处理。 首先，通过记忆存储操作将经验写入原生记忆。

The original trajectory prefix is then replayed with memory reading disabled. After the designated breakpoint, memory reading is enabled, and the model is trained to predict the successful continuation conditioned on the stored experience. The experience text and trajectory prefix are excluded from the training loss, while the subsequent assistant actions provide the supervision signal.

然后，在禁用记忆读取的情况下回放原始轨迹前缀。 到达指定断点后启用记忆读取，训练模型以已存储经验为条件，预测成功续接轨迹。 经验文本和轨迹前缀不计入训练损失，后续助手动作提供监督信号。

We investigate both cross-entropy supervision and knowledge distillation. For knowledge distillation, the frozen backbone teacher receives the experience as text, while the student accesses the same experience only through native memory. Both models use teacher forcing on the same task history and reference continuation. We minimize the forward KL divergence from the teacher distribution to the student distribution over supervised assistant tokens.

我们研究交叉熵监督和知识蒸馏两种方式。 对于知识蒸馏，冻结的骨干教师接收文本形式的经验，学生则仅通过原生记忆访问相同经验。 两个模型在相同任务历史和参考续接轨迹上，均使用教师强制。 我们在受监督助手 token 上，最小化从教师分布到学生分布的前向 KL 散度。

We additionally investigate two supervision scopes. The First-Step setting supervises only the first assistant turn following the breakpoint, and the Full-Suffix setting supervises the complete successful continuation. We use knowledge distillation with Full-Suffix supervision for the application-specific post-training of MemT-1.0-Nano and evaluate the resulting checkpoint in the main experiments. Appendix E details the sample serialization and supervision masks.

此外，我们研究两种监督范围。 First-Step 设置只监督断点后的第一个助手轮次，Full-Suffix 设置监督完整的成功续接轨迹。 在 MemT-1.0-Nano 面向应用的后训练中，我们采用 Full-Suffix 监督的知识蒸馏，并在主要实验中评估所得检查点。 附录 E 详述了样本序列化与监督掩码。

7.4. Agentic Coding Evaluation

We evaluate MemT-1.0-Nano on two agentic coding benchmarks, including SWE-bench Verified, containing 500 tasks, and SWE-bench Multilingual, containing 300 tasks across nine programming languages. Both benchmarks require agents to resolve real-world software issues through repository exploration, code modifications, and executable tests. All baselines use Qwen3.6-35B-A3B as the backbone and follow the official SWE-bench verification protocol.

我们在两个智能体编程基准上评估 MemT-1.0-Nano：包含 500 个任务的 SWE-bench Verified，以及包含九种编程语言、300 个任务的 SWE-bench Multilingual。 两个基准均要求智能体通过仓库探索、代码修改和可执行测试，解决真实软件问题。 所有基线都使用 Qwen3.6-35B-A3B 作为骨干，并遵循官方 SWE-bench 验证协议。

We compare Qwen3.6-35B-A3B (No Experience), Qwen3.6-35B-A3B (Textual Experience), and MemT-1.0-Nano (Native Memory). The No Experience baseline measures the benefit of experience reuse, while Textual Experience provides a reference for evaluating its effectiveness through native memory. The Harness-Triggered setting uses the same breakpoint-detection logic as training data construction. The Before Execution setting is used to assess robustness to experience injection timing. Each task is executed three times independently.

我们比较 Qwen3.6-35B-A3B（No Experience）、Qwen3.6-35B-A3B（Textual Experience）和 MemT-1.0-Nano（Native Memory）。 No Experience 基线 衡量经验复用的收益，而 Textual Experience 为评估通过原生记忆利用经验的有效性提供参考。 Harness-Triggered 设置采用与训练数据构建相同的断点检测逻辑。 Before Execution 设置用于评估对经验注入时机的稳健性。 每项任务独立执行三次。

As shown in Table 5, we report the average success rate, pass@3, representing the proportion of tasks solved at least once across the three runs, and pass^3, representing the proportion of tasks solved in all three runs. MemT-1.0-Nano consistently improves the average success rate and repeated-run reliability over the backbone without experience. On SWE-bench Verified, in the Before Execution setting, MemT-1.0-Nano achieves an average success rate of 69.33%, improving upon the No Experience baseline of 66.07%. This slightly exceeds the 68.87% achieved by Textual Experience. Moreover, pass^3 increases from 54.00% to 61.60%, matching the 61.60% achieved by Textual Experience. These results indicate that native memory can preserve the effectiveness of procedural experience without requiring the corresponding experience text to remain in the task context.

如 表5 所示，我们报告平均成功率、表示三次运行中至少解决一次的任务比例的 pass@3，以及表示三次运行全部成功的任务比例的 pass^3。 相较不使用经验的骨干，MemT-1.0-Nano 始终提高了平均成功率和重复运行可靠性。 在 SWE-bench Verified 的 Before Execution 设置中，MemT-1.0-Nano 的平均成功率达到 69.33%，高于 No Experience 基线的 66.07%。 这也略高于 Textual Experience 的 68.87%。 此外，pass^3 从 54.00% 升至 61.60%，与 Textual Experience 的 61.60% 持平。 这些结果表明，原生记忆能够保留程序性经验的有效性，而无需将相应经验文本保留在任务上下文中。

表5：两种经验注入设置下，在 SWE-bench Verified（500 个任务）和 SWE-bench Multilingual（300 个任务）上的智能体编程性能。加粗和下划线分别表示最高和次高的不同得分。

Benchmark	Store Timing	Method	pass^3	Avg. Success	pass@3
Verified	Before Execution	Qwen3.6-35B-A3B (No Experience)	54.00	66.07	77.60
		Qwen3.6-35B-A3B (Textual Experience)	61.60	68.87	76.40
		MemT-1.0-Nano (Native Memory)	61.60	69.33	77.40
	Harness-Triggered	Qwen3.6-35B-A3B (No Experience)	54.00	66.07	77.60
		Qwen3.6-35B-A3B (Textual Experience)	61.80	69.13	76.80
		MemT-1.0-Nano (Native Memory)	62.80	69.73	77.80
Multilingual	Before Execution	Qwen3.6-35B-A3B (No Experience)	45.33	60.00	73.67
		Qwen3.6-35B-A3B (Textual Experience)	56.33	64.22	72.67
		MemT-1.0-Nano (Native Memory)	53.67	62.89	72.00
	Harness-Triggered	Qwen3.6-35B-A3B (No Experience)	45.33	60.00	73.67
		Qwen3.6-35B-A3B (Textual Experience)	53.67	63.00	72.67
		MemT-1.0-Nano (Native Memory)	55.33	63.56	73.00

所有数值均为百分比，每项任务独立运行三次。pass^3 表示三次均成功的任务比例；Avg. Success 表示各次运行的平均成功率；pass@3 表示至少成功一次的任务比例。

In the Harness-Triggered setting, MemT-1.0-Nano further improves the average success rate to 69.73%, compared with 66.07% for No Experience and 69.13% for rank-1 textual experience. The corresponding pass^3 reaches 62.80%, representing an improvement of 8.80 percentage points over the backbone and 1.00 percentage point over fixed rank-1 textual experience. This suggests that native memory remains effective when experience is introduced dynamically during agent execution, rather than only at task initialization.

在 Harness-Triggered 设置中，MemT-1.0-Nano 将平均成功率进一步提高至 69.73%，而 No Experience 为 66.07%，排名第 1 的文本经验为 69.13%。 对应的 pass^3 达到 62.80%，比骨干高 8.80 个百分点，比固定使用排名第 1 的文本经验高 1.00 个百分点。 这表明，经验在智能体执行期间动态引入时，原生记忆仍然有效，其作用并不限于任务初始化时注入经验。

We observe consistent improvements on SWE-bench Multilingual. In the Before Execution setting, MemT-1.0-Nano improves the average success rate from 60.00% to 62.89% and pass^3 from 45.33% to 53.67%. In the Harness-Triggered setting, the average success rate reaches 63.56%, exceeding the fixed rank-1 textual experience result of 63.00%. These findings provide further evidence that experience internalization generalizes across coding tasks involving different repositories and programming languages.

我们在 SWE-bench Multilingual 上也观察到一致提升。 在 Before Execution 设置中，MemT-1.0-Nano 将平均成功率从 60.00% 提高至 62.89%，将 pass^3 从 45.33% 提高至 53.67%。 在 Harness-Triggered 设置中，平均成功率达到 63.56%，超过固定使用排名第 1 的文本经验所得的 63.00%。 这些发现进一步表明，经验内化能够泛化到不同仓库和编程语言的编程任务。

The gains are most pronounced in repeated-run reliability, suggesting that stored experience helps agents solve tasks more consistently. Although pass^3 improves substantially, pass@3 does not consistently increase. The comparable behavior of MemT-1.0-Nano and textual experience further suggests that procedural guidance can be represented and utilized through native memory.

重复运行可靠性方面的收益最明显，表明已存储经验能够帮助智能体更一致地解决任务。 尽管 pass^3 显著提高，pass@3 并未始终增加。 MemT-1.0-Nano 与文本经验相近的表现进一步说明，程序性指导可以通过原生记忆表示和利用。

We also evaluate several training configurations on a 100-task development subset of SWE-bench Verified. Detailed comparisons and additional experimental results are provided in Appendix E.

我们还在 SWE-bench Verified 的一个包含 100 个任务的开发子集上评估了若干训练配置。 详细比较和额外实验结果见 附录 E。

7.5. Case Study and Discussion

To better understand how native memory affects agent behavior, we illustrate a representative task from SWE-bench Verified involving the two-digit year parsing logic in Django's parse_http_date function.

为更好地理解原生记忆如何影响智能体行为，我们展示 SWE-bench Verified 中的一个代表性任务，该任务涉及 Django 的 parse_http_date 函数中两位数年份的解析逻辑。

We compare execution trajectories under the No Experience, Textual Experience, and Native Memory settings. Without experience, the agent modifies the relevant parsing logic shortly after locating the implementation, introducing targeted reproduction and verification steps later in the execution. In contrast, agents in both the Textual Experience and Native Memory settings construct reproduction tests before applying modifications. In the MemT-1.0-Nano trajectory, the agent further revisits a boundary condition after its initial modification, corrects the implementation, and performs additional verification.

我们比较 No Experience、Textual Experience 和 Native Memory 三种设置下的执行轨迹。 没有经验时，智能体在定位实现后不久便修改相关解析逻辑，在执行后期才加入有针对性的复现与验证步骤。 相比之下，Textual Experience 和 Native Memory 设置下的智能体，都会先构建复现测试，再应用修改。 在 MemT-1.0-Nano 的轨迹中，智能体还在初次修改后重新检查一个边界条件，纠正实现，并进行额外验证。

These observations show how stored procedural experience can guide task execution without appearing in textual context. Together with the benchmark improvements, they demonstrate that MemT-1.0 can reuse procedural experience to improve agentic task performance. This supports its potential for developing self-evolving agents: experience from past interactions can be stored in native memory and applied to new tasks.

这些观察展示了已存储的程序性经验如何在不出现在文本上下文中的情况下，指导任务执行。 结合基准上的提升，它们说明 MemT-1.0 能够通过复用程序性经验，改善智能体任务性能。 这支持其发展自我演化智能体的潜力：过往交互中的经验可以存入原生记忆，并应用于新任务。

Several challenges remain. The current framework relies on externally constructed experience, making its effectiveness dependent on experience quality and retrieval. Storing experience also introduces computation, so reduced textual context does not necessarily imply lower overall execution costs. Extending experience reuse into an autonomous continual learning process remains an open direction.

仍有若干挑战有待解决。 当前框架依赖外部构建的经验，其有效性取决于经验质量和检索效果。 存储经验也引入计算，因此减少文本上下文不一定意味着整体执行成本降低。 将经验复用扩展为自主的持续学习过程，仍是开放方向。

Future work will explore the co-evolution of the harness and MemT-1.0 through continued interaction with the environment. The harness can improve experience acquisition and selection, while MemT-1.0 learns to reuse that experience more effectively. Together, these advances could support continual learning in self-evolving agents for long-horizon tasks.

未来工作将探索执行框架与 MemT-1.0 通过持续环境交互进行协同演化。 执行框架可以改善经验获取和选择，而 MemT-1.0 则学习更有效地复用这些经验。 这些进展相互结合，有望支持长程任务中自我演化智能体的持续学习。

8. Limitations and Discussion

Despite the progress of MemT-1.0 in native memory, it still has several limitations. The fixed-size memory state imposes a capacity constraint. For the fixed memory, MemT-1.0 maintains an $O(1)$ state with respect to the interaction history length $N$. Compared with methods whose stored representations grow as $O(N)$, this bounded state has limited capacity to preserve detailed historical information.

尽管 MemT-1.0 在原生记忆方面取得进展，仍存在若干局限。 固定大小的记忆状态带来了容量约束。 对于固定记忆，MemT-1.0 相对于交互历史长度 $N$，维护一个 $O(1)$ 的状态。 与已存储表示按 $O(N)$ 增长的方法相比，这种有界状态保留详细历史信息的容量有限。

As interactions accumulate, new information must be incorporated into an existing representation, creating trade-offs between retaining earlier details and adding recent information. The resulting loss of memory fidelity remains a challenge for tasks requiring precise recall. At the same time, a fixed-size state offers opportunities beyond reducing storage. Each update incorporates new information into a representation, which has already been shaped by earlier interactions.

随着交互累积，新信息必须纳入已有表示，从而在保留早期细节与加入近期信息之间形成权衡。 由此产生的记忆保真度损失，仍是需要精确回忆的任务所面临的挑战。 同时，固定大小状态所带来的机会不止于减少存储。 每次更新都将新信息融入一个已被更早交互塑造的表示。

This enables successive compression of accumulated history. Compared with maintaining a growing collection of historical records, this recurrent organization may provide a basis for latent reflection. It potentially allows iterative consolidation of experience within the memory state. Such a mechanism could support online self-evolution by allowing accumulated experience to influence subsequent behavior.

这使累积历史能够得到连续压缩。 与维护不断增长的历史记录集合相比，这种循环组织方式可能为潜在空间中的反思提供基础。 它有可能支持在记忆状态内迭代整合经验。 这种机制使累积经验影响后续行为，从而可能支持在线自我演化。

Finally, the relationship between local memory and hyper memory also deserves further study. Local memory represents information accumulated during interaction, while hyper memory encodes learned capabilities for storing and utilizing that information. An important question is whether experience accumulated in local memory can be consolidated into hyper-memory parameters through further training, and how knowledge encoded in hyper memory can be instantiated and adapted through local memory. Therefore, memory capability itself can be viewed as a higher-order form of learned information. Exploring this relationship may help connect adaptation within individual interactions with improvements in memory capability across tasks.

最后，局部记忆与超记忆之间的关系也值得进一步研究。 局部记忆表示交互中积累的信息，超记忆则编码通过学习获得的存储和利用这些信息的能力。 一个重要问题是：局部记忆中积累的经验能否通过进一步训练整合到超记忆参数中，以及超记忆中编码的知识如何通过局部记忆实例化和适应。 因此，记忆能力本身也可以视为一种更高阶的已学习信息。 探索这一关系，可能有助于将单次交互内的适应，与跨任务记忆能力的提升联系起来。

9. Conclusion

In this report, we presented MemT-1.0, a family of memory foundation models that integrates persistent memory into model computation. Through the token-level stable memory updates, Mixture-of-Storage, a deliberate backbone preparation, and a two-stage training pipeline, MemT-1.0 supports efficient storage and utilization of historical information. Both models outperform the evaluated parametric memory baselines on average, while largely preserving the general capabilities of their respective backbones under the noise setting. Applications to coding agents further demonstrate that native memory can internalize procedural experience to improve task success and execution consistency. Future work will focus on improving long-term memory, scaling memory capacity, and reducing the cost of memory updates. We will explore autonomous experience acquisition and continual memory consolidation across tasks, moving toward agents learning through memory-native evolution.

本报告介绍了 MemT-1.0，这是一组将持久记忆融入模型计算的记忆基础模型。 通过 token 级稳定记忆更新、混合存储、精心的骨干模型准备和两阶段训练流程，MemT-1.0 支持高效存储与利用历史信息。 两个模型的平均表现均优于所评估的参数化记忆基线，同时在噪声设置下基本保留了各自骨干的通用能力。 在编程智能体中的应用进一步表明，原生记忆能够内化程序性经验，从而提升任务成功率与执行一致性。 未来工作将聚焦于改善长期记忆、扩展记忆容量，并降低记忆更新成本。 我们将探索自主经验获取和跨任务持续记忆整合，迈向通过记忆原生演化进行学习的智能体。