MEMENTO: Teaching LLMs to Manage Their Own Context

MemoryReasoningContext CompressionRLCOLM 2026微软研究院

Kontonis V, Zeng Y, Garg S, et al. MEMENTO: Teaching LLMs to Manage Their Own Context. 2026.
https://github.com/microsoft/memento
https://huggingface.co/datasets/microsoft/OpenMementos

MEMENTO：教会大语言模型管理自身上下文

Abstract

Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state. We introduce Memento: a method that teaches models to segment reasoning into blocks, compress each block into a memento, i.e., a dense state summary, and reason forward by attending only to mementos, reducing context, KV cache, and compute. To train Memento models, we release OpenMementos, a public dataset of 228K reasoning traces derived from OpenThoughts-v3, segmented and annotated with intermediate summaries. We show that a two-stage SFT recipe on OpenMementos is effective across different model families (Qwen3, Phi-4, Olmo 3) and scales (8B–32B parameters).

推理模型在冗长、非结构化的思维流中思考，缺乏压缩或组织自身中间状态的机制。我们提出 Memento：一种教会模型将推理划分为若干块、把每一块压缩成 memento（即紧凑的状态摘要），并且仅关注这些 memento 继续推理的方法，从而减少上下文、KV 缓存与计算量。为训练 Memento 模型，我们发布了 OpenMementos，这是一个包含 228K 条推理轨迹的公开数据集，源自 OpenThoughts-v3，经过分块并标注了中间摘要。我们表明，在 OpenMementos 上进行两阶段监督微调（SFT）的训练方案，适用于不同模型家族（Qwen3、Phi-4、Olmo 3）和规模（8B–32B 参数）。

Trained models maintain strong accuracy on math, science, and coding benchmarks while achieving ∼2.5× peak KV cache reduction. We extend vLLM to support our inference method, achieving ∼1.75× throughput improvement while also enabling us to perform RL and further improve accuracy. Finally, we identify a dual information stream: information from each reasoning block is carried both by the memento text and by the corresponding KV states, which retain implicit information from the original block. Removing this channel drops accuracy by 15 pp on AIME24.

训练后的模型在数学、科学和编程基准上保持较强的准确率，同时将峰值 KV 缓存降至原来的约 1/2.5。我们扩展了 vLLM 以支持这一推理方法，使吞吐量达到原来的约 1.75 倍，同时支持通过强化学习（RL）进一步提升准确率。最后，我们发现了一种 双重信息流：每个推理块的信息同时由 memento 文本和相应的 KV 状态承载，后者保留了原始块中的隐式信息。移除这一通道会使 AIME24 上的准确率下降 15 个百分点。

Memento 图0：overview — 图0：**Memento 概览。** **左上：** SFT 数据生成流程。从一条推理轨迹出发，先将文本拆分成句子，使用 LLM 对每个句子边界作为潜在停止位置的适合程度评分，再通过算法优化边界选择，最后使用 LLM 将每个块概括成一个 memento。**右上：** 推理时的稀疏注意力。模型交替生成思考块（ $T_{i}$ ）和 memento（ $M_{i}$ ）；一旦生成 memento，就会物理删除前一个思考块的 KV 缓存条目。锯齿状 KV 轨迹展示了由此形成的内存变化模式。**左下：** Qwen2.5-7B-Instruct 在 AIME25 上的数据规模实验；在不同数据量下，Memento 与使用 OpenThoughts 的普通 SFT 呈现类似的扩展趋势（图3）。**中下：** 三种模型在 AIME26、GPQA-D 和 LCB 上的准确率；浅色柱表示 RL 为 Qwen3-8B 带来的增益（第 5 节）。**右下：** 对所有基准类别取平均的峰值 KV 缓存（GB），降至原来的约 1/2–1/2.5。

1. Introduction

Large language models routinely reason at test time, spending thousands of tokens working through a problem before arriving at an answer. This has led to dramatic gains on hard reasoning benchmarks, but has also created a new problem: reasoning models have no built-in mechanism to organize their chain-of-thought. A 32K-token CoT is a flat, unstructured stream, and there is no mechanism for the model to mark an intermediate result as worth keeping, or to compress a long derivation into a compact conclusion that it can reference later. Every past token sits in the attention window at equal cost, and the model has learned no way to drop it.

大语言模型通常会在测试时进行推理，在得出答案前花费数千个 token 来求解问题。这显著提升了模型在高难度推理基准上的表现，但也带来了一个新问题：推理模型没有内置机制来组织自己的思维链。一条包含 32K 个 token 的思维链（CoT）是一段平铺、非结构化的信息流，模型没有机制将某个中间结果标记为值得保留，也无法把冗长的推导压缩为可供后续引用的简洁结论。每个历史 token 都以同等成本占据注意力窗口，而模型尚未学会如何舍弃它们。

We propose Memento, an approach that trains models to segment their chain of thought into semantically coherent blocks and, after each block, generate a compressed summary that we call a memento. Rather than a summary in the usual expository sense, each memento is a minimal record of a reasoning block, preserving its conclusions, intermediate values, and key directional decisions in as few tokens as possible. Once a memento is produced, the preceding thinking block is masked within a single, uninterrupted generation call via a custom vLLM based engine: at every subsequent step, the model attends only to past mementos and the current block. Each block is compressed to a ∼5–20× smaller size on average, so the effective context the model attends to is a fraction of the full trace. The inference procedure is illustrated in Figure 0 (right).

我们提出 Memento，通过训练让模型将思维链划分为语义连贯的块，并在每个块之后生成一个压缩摘要，我们将其称为 memento。与通常用于解释说明的摘要不同，每个 memento 都是对一个推理块的最小记录，用尽可能少的 token 保留其结论、中间数值以及关键的方向性决策。一旦生成 memento，定制的 vLLM 引擎就会在一次连续、不间断的生成调用内屏蔽前一个思考块：在随后的每一步中，模型只关注历史 memento 和当前块。每个块平均被压缩到原来的约 1/5–1/20，因此模型所关注的有效上下文仅为完整轨迹的一小部分。推理过程如图0 右侧所示。

Crucially, because masking happens in-place rather than by restarting generation, the KV cache entries of each memento are computed while the full block is still in context and retained after the block is masked. Despite the fact that the original thinking tokens are gone, they remain implicitly present in the representations of the memento KV states. This creates a dual information stream: the explicit memento text plus an implicit representational channel through the cached KV states. We verify this experimentally: recomputing memento KVs without block context reduces accuracy by 15 pp on AIME'24 (Section 6.2.1), and our probing experiments (Section 6.2.2) demonstrate that block information not present in the memento text is still recoverable from the memento KV states, with upper layers carrying the most task-relevant signal.

关键在于，屏蔽操作是在原位完成的，而不是通过重启生成来实现，因此每个 memento 的 KV 缓存条目是在完整推理块仍处于上下文中时计算的，并在该块被屏蔽后继续保留。尽管原始思考 token 已经消失，它们仍隐式存在于 memento 的 KV 状态表示中。这形成了一种 双重信息流：显式的 memento 文本，以及通过缓存 KV 状态传递的隐式表示通道。我们通过实验验证了这一点：在没有推理块上下文的情况下重新计算 memento 的 KV，会使 AIME'24 准确率下降 15 个百分点（第 6.2.1 节）；探针实验（第 6.2.2 节）还表明，memento 文本中没有出现的块信息仍可从其 KV 状态中恢复，而且更高层承载了最多的任务相关信号。

A prime concern is that compression could destroy reasoning capacity. Our experiments across three model families, i.e., Qwen3 (8B/32B), Phi-4-reasoning (14B), and Olmo-3-7B-Think, show that this is not the case. On AIME'26, Qwen3-32B with Memento loses just 2.6 pp (72.6% vs. 75.2%) while cutting peak KV cache by ∼2×. Averaged across the five benchmark groups in Table 1, the accuracy gap is 3.5 pp at 32B and 6.3 pp at 8B, and we observe that the gap shrinks with scale within the same model family, suggesting that larger models manage compressed context more effectively. Further, we show that RL fine-tuning can close the remaining gap, enabled by native block masking support in our vLLM fork.

一个主要顾虑是，压缩可能破坏推理能力。我们在三个模型家族——Qwen3（8B/32B）、Phi-4-reasoning（14B）和 Olmo-3-7B-Think——上的实验表明，事实并非如此。在 AIME'26 上，采用 Memento 的 Qwen3-32B 准确率仅下降 2.6 个百分点（72.6%，原模型为 75.2%），同时将峰值 KV 缓存减半。对表1 中的五组基准取平均，32B 模型的准确率差距为 3.5 个百分点，8B 模型则为 6.3 个百分点；我们还观察到，同一模型家族内的差距随规模增大而缩小，这说明更大的模型能更有效地管理压缩后的上下文。此外，借助我们修改的 vLLM 对块屏蔽的原生支持，我们展示了 RL 微调能够弥合剩余差距。

To train Memento models, we construct OpenMementos, a public dataset of 228K segmented and summarized reasoning traces derived from OpenThoughts. Building this dataset required solving a non-trivial annotation problem: reasoning traces lack natural segment boundaries, and naive summarization loses the precise intermediate state a model needs to continue. Our pipeline combines LLM-scored boundary detection, algorithmic segmentation, and iterative judge-refined summarization to produce training data where each memento is both faithful and minimal. Our key contributions can be summarized as follows:

为训练 Memento 模型，我们构建了 OpenMementos，这是一个包含 228K 条分块并附有摘要的推理轨迹的公开数据集，源自 OpenThoughts。构建这个数据集需要解决一个并不简单的标注问题：推理轨迹缺少自然的分段边界，而朴素的摘要方法会丢失模型继续推理所需的精确中间状态。我们的流程结合了 LLM 评分的边界检测、算法分段以及借助评审反馈迭代改进的摘要生成，从而得到每个 memento 都忠实而精简的训练数据。我们的主要贡献可以概括如下：

OpenMementos: a 228K-trace public dataset of segmented, summarized reasoning chains, with the annotation pipeline and code.
Demonstration at scale across three model families (Qwen3 8B/32B, Phi-4-reasoning 14B, Olmo-3-7B-Think), showing that models internalize summarization as a learned capability while preserving reasoning accuracy at 2–3× peak KV cache reduction.
Native block masking in vLLM, a custom fork that supports in-place KV cache masking within a single generation call: a key infrastructure bottleneck for both inference and training with Memento. This enables RL fine-tuning with block masking, which allows us to close the accuracy gap.
The dual information stream: we identify and verify that memento KV states encode information from masked blocks which is a mechanism absent in restart-based approaches. Removing this channel drops accuracy by 15 pp on AIME'24.

OpenMementos：一个包含 228K 条推理轨迹的公开数据集，推理链经过分块并配有摘要，同时提供标注流程和代码。
跨规模验证：在三个模型家族（Qwen3 8B/32B、Phi-4-reasoning 14B、Olmo-3-7B-Think）上展示，模型能将摘要生成内化为习得的能力，在保持推理准确率的同时，将峰值 KV 缓存降至原来的 1/2–1/3。
vLLM 中的原生块屏蔽：我们定制了支持在单次生成调用内原位屏蔽 KV 缓存的 vLLM 版本，解决了 Memento 推理与训练共同面临的一项关键基础设施瓶颈。这使 带块屏蔽的 RL 微调 成为可能，从而弥合准确率差距。
双重信息流：我们发现并验证，memento 的 KV 状态会编码被屏蔽块的信息，而基于重启的方法不具备这一机制。移除该通道会使 AIME'24 的准确率下降 15 个百分点。

Context management for long-running models is typically handled through external infrastructure: separate summarizers, memory modules, or orchestration logic. We instead focus on teaching models to manage their own context during reasoning, as an internal capability rather than an external system. Among works that do train models for context management, MemAgent reads text in segments and updates a fixed-size memory via an overwrite strategy trained with RL, and MEM1 takes a similar approach for multi-turn agent interactions, maintaining a compact internal state across tool calls and environment observations. Both primarily focus on managing external information, retrieved documents, tool outputs, and environment observations, rather than complex reasoning chains typically observed in models solving hard math or coding problems.

长时间运行的模型通常通过外部基础设施管理上下文，例如独立的摘要器、记忆模块或编排逻辑。我们则着重教会模型在推理过程中管理自身上下文，使之成为模型的内部能力，而不是外部系统的功能。在同样通过训练实现上下文管理的工作中，MemAgent 分段读取文本，并通过 RL 训练的覆写策略更新固定大小的记忆；MEM1 则将类似方法用于多轮智能体交互，在工具调用和环境观测之间维护紧凑的内部状态。两者主要关注外部信息、检索文档、工具输出和环境观测的管理，而不是模型解决高难度数学或编程问题时通常产生的复杂推理链。

The most closely related works to ours train models to compress their own reasoning output in domains such as math: InftyThink, InftyThink+, Accordion-Thinking, and The Markovian Thinker. These works break reasoning into (potentially variable-length) chunks and train the model to continue from a compact textual carryover rather than the full prior chunk. InftyThink relies on SFT alone, while InftyThink+, Accordion-Thinking, and The Markovian Thinker additionally use RL to improve this chunk-to-chunk carryover. All of these works operate at the text level: after each reasoning chunk, the future context is rebuilt from compact text alone, discarding the original reasoning tokens and their KV cache representations.

与我们最接近的工作会训练模型压缩其在数学等领域生成的推理内容，包括 InftyThink、InftyThink+、Accordion-Thinking 和 The Markovian Thinker。这些方法将推理拆分为可能长度不一的片段，并训练模型依据精简的文本承接信息继续推理，而不是依赖完整的前一片段。 InftyThink 仅依赖 SFT，而 InftyThink+、Accordion-Thinking 和 The Markovian Thinker 还使用 RL 来改善片段之间的信息传递。这些工作都在文本层面运作：每个推理片段结束后，仅通过精简文本重建后续上下文，丢弃原始推理 token 及其 KV 缓存表示。

Memento differs in that it retains summary KV entries via in-engine attention masking rather than text-level context rebuilding, creating a dual stream of information: the explicit memento text, and the implicit representations encoded in the memento's KV cache. Our experiments demonstrate that useful information is stored in these KV entries—dropping them and recomputing memento KVs without block context reduces AIME24 accuracy by 15 percentage points (Section 6.2.1).

Memento 的不同之处在于，它通过引擎内部的注意力屏蔽保留摘要的 KV 条目，而不是在文本层面重建上下文，从而形成双重信息流：显式的 memento 文本，以及编码在 memento 的 KV 缓存中的隐式表示。我们的实验表明，这些 KV 条目储存了有用信息：丢弃它们，并在没有推理块上下文的情况下重新计算 memento 的 KV，会使 AIME24 准确率下降 15 个百分点（第 6.2.1 节）。

Another related work (PENCIL) explores learned context management for models trained from scratch on synthetic tasks. PENCIL teaches models to erase intermediate reasoning via reduction rules, enabling small (25M parameter) models to solve 3-SAT problems and Einstein's Puzzle (a multi-constraint logic deduction) with 2K context length. PENCIL demonstrates that the potential of context management extends beyond memory and throughput efficiency; it can enable models to solve significantly harder problems than standard CoT permits. Whether such gains can be achieved at larger scale and in settings such as math and coding problems, for instance through Memento-style compression, is an interesting direction for future work.

另一项相关工作 PENCIL 探索了在合成任务上从头训练的模型如何学习上下文管理。 PENCIL 通过归约规则教会模型擦除中间推理，使小型模型（25M 参数）能够在 2K 上下文长度内解决 3-SAT 问题和爱因斯坦谜题（一种多约束逻辑推理问题）。 PENCIL 表明，上下文管理的潜力不止于内存和吞吐效率；它能够让模型解决远比标准 CoT 所能处理的问题更难的任务。能否在更大规模、以及数学和编程等问题场景中获得类似收益，例如通过 Memento 式压缩来实现，是值得未来研究的方向。

A closely related line of work compresses reasoning chunks into learned gist tokens, i.e., special-purpose tokens whose KV cache entries encode a compressed representation of the preceding chunk, after which the original tokens are evicted. A limitation of gist-token approaches is that interpretability is lost: the compressed state is encoded entirely in hidden representations. Memento's summaries are natural-language text, preserving interpretability while still achieving compression.

另一条密切相关的研究路线将推理片段压缩为习得的 gist token，即专门用途的 token，其 KV 缓存条目编码前一片段的压缩表示，随后移除原始 token。 gist token 方法的一个局限在于丧失可解释性：压缩状态完全编码在隐藏表示中。 Memento 的摘要采用自然语言文本，在实现压缩的同时保留了可解释性。

A complementary line of work aims to reduce the memory footprint of reasoning through shorter traces or direct KV cache compression. These include methods that train models to skip low-importance tokens, train on compressed reasoning traces, or steer the model to produce shorter traces via RL. Other works replace explicit reasoning tokens with latent representations. At the KV cache level, inference-time methods such as ThinKV, R-KV, LazyEviction, and Reasoning Path Compression prune or quantize cache entries based on attention patterns, while architectures such as sliding-window attention limit the attention span by design. These architecture-level approaches are orthogonal to Memento and many can likely compose with it: for example, we show that Olmo-3-7B-Think that has sliding-window attention can be effectively combined with Memento.

一条互补的研究路线旨在通过缩短推理轨迹或直接压缩 KV 缓存，降低推理的内存占用。相关方法包括训练模型跳过低重要性 token、使用压缩后的推理轨迹训练模型，以及通过 RL 引导模型生成更短的轨迹。另一些工作则用潜在表示替代显式推理 token。在 KV 缓存层面，ThinKV、R-KV、LazyEviction 和 Reasoning Path Compression 等推理时方法依据注意力模式裁剪或量化缓存条目，而滑动窗口注意力等架构则从设计上限制注意力跨度。这些架构层面的方法与 Memento 是相互独立的，许多方法也很可能与之结合：例如，我们展示了采用滑动窗口注意力的 Olmo-3-7B-Think 可以与 Memento 有效结合。

Finally, we note a naming overlap with recent work on memory-augmented agents: Zhou et al. (2025) and Wang (2026) also use the name Memento for systems that maintain external episodic memory for inference-time agent adaptation.

最后，我们指出，它与近期记忆增强智能体工作存在命名重叠：Zhou 等人（2025）和 Wang（2026）也使用 Memento 这个名称，指代通过维护外部情景记忆实现推理时智能体适应的系统。

3. OpenMementos Dataset

Training models to simultaneously reason and manage context requires high-quality annotated data: reasoning traces segmented into semantically coherent blocks, paired with dense summaries. A core challenge is that typical reasoning traces are not a sequence of independent thoughts; they are a continuous stream without “natural” boundaries. Here we describe our data generation pipeline (Figure 0, top left), which takes raw CoT traces and produces structured traces annotated with mementos.

训练模型同时进行推理和上下文管理，需要高质量的标注数据：将推理轨迹划分为语义连贯的块，并为其配上紧凑摘要。一个核心挑战是，典型推理轨迹并非一连串相互独立的思考，而是没有“自然”边界的连续信息流。下面介绍我们的数据生成流程（图0 左上），它以原始 CoT 轨迹为输入，生成带有 memento 标注的结构化轨迹。

Design rationale. Each stage in our pipeline reflects a deliberate design choice. Early on, we tried having a frontier LLM directly segment CoTs into semantically coherent blocks. This failed. Even strong models struggle with a combinatorial optimization problem that considers all possible partitions, requiring simultaneous reasoning about block coherence, size balance, and semantic boundaries.

设计依据。 流程中的每个阶段都体现了一项经过斟酌的设计选择。早期，我们尝试让前沿 LLM 直接将 CoT 划分为语义连贯的块。这一尝试失败了。即使是很强的模型，也难以处理需要考虑所有可能划分的组合优化问题，因为它要求同时推理块内连贯性、大小均衡以及语义边界。

To simplify, we factored the problem: boundary scoring asks a local question (“is this a good place to slice the CoT?”), which LLMs handle well, while the global optimization of boundary selection given the LLM scores is handled algorithmically. We then use an LLM judge to grade the quality of summarization. Defining “good summary” programmatically is difficult, but LLMs can give reasonable scores against explicit rubrics. We then chose iterative refinement of mementos over single-shot summarization as initial mementos often miss key formulas or intermediate values; a zero-shot approach achieves only 28% pass rate (scoring $\geq 8 / 10$ on our rubric), while the judge-feedback loop brings this to 92%.

为简化任务，我们将问题分解：边界评分只需回答局部问题——“这里是否适合切分 CoT？”，这是 LLM 擅长的；而给定 LLM 评分后的边界选择全局优化，则由算法处理。随后，我们使用 LLM 评审器评估摘要质量。用程序定义“好摘要”很困难，但 LLM 能够依据明确的评分标准给出合理分数。我们进一步选择对 memento 进行迭代改进，而非单次摘要生成，因为初始 memento 常常遗漏关键公式或中间数值；零样本方法的通过率仅为 28%（按我们的标准达到 ≥ 8/10 分），而评审反馈循环能将其提高到 92%。

Stage 0: Seed selection. We source 228K reasoning traces from OpenThoughts-v3, a widely-adopted dataset of CoT traces generated by QwQ-32B, and process them through our annotation pipeline to produce the final OpenMementos dataset. While we could regenerate traces using stronger teachers, we leverage OpenThoughts because: (1) substantial effort has already been invested in generating these traces at scale; (2) the OpenThinker-3 paper provides extensive baselines, making it an ideal testbed; and (3) our hypothesis that traces from a relatively strong reasoner (QwQ-32B) should transfer across model families is confirmed empirically (works for Qwen3, Phi-4, Olmo 3).

阶段 0：种子选择。 我们从 OpenThoughts-v3 获取 228K 条推理轨迹，并通过标注流程处理，生成最终的 OpenMementos 数据集；OpenThoughts-v3 是一个广泛采用的 CoT 轨迹数据集，由 QwQ-32B 生成。尽管可以用更强的教师模型重新生成轨迹，我们仍采用 OpenThoughts，原因有三点：（1）大规模生成这些轨迹已经投入了大量工作；（2）OpenThinker-3 论文提供了丰富的基线，使其成为理想的试验平台；（3）实验证实了我们的假设，即较强推理模型 QwQ-32B 的轨迹能够跨模型家族迁移，对 Qwen3、Phi-4 和 Olmo 3 均有效。

Stage 1: Sentence splitting. We partition reasoning traces into atomic “sentences”: complete, modular thoughts that can stand alone. Code blocks and multi-line math are detected and protected as atomic units. Plain text is split at sentence boundaries (avoiding splits inside parentheses, inline math, or abbreviations). Finally, we merge logically connected fragments: sentences ending with colons are attached to the next; continuation words (Therefore, Thus, So) signal a need to merge with the preceding sentence; short fragments (<5 tokens) and consecutive math expressions are consolidated. This structure-aware splitting reduces candidate boundaries by ∼2× vs. naive sentence splitting (397 → 187 per trace on average).

阶段 1：分句。 我们将推理轨迹拆分为原子的“句子”，即可以独立成立、完整且模块化的思考单元。代码块和多行数学表达式会被识别并作为原子单元保护起来。普通文本按句子边界拆分，同时避免在括号、行内公式或缩写内部切分。最后，我们合并逻辑上相连的片段：以冒号结尾的句子与下一句相连；Therefore、Thus、So 等承接词表明需要与前一句合并；不足 5 个 token 的短片段和连续数学表达式也会被合并。与朴素分句相比，这种考虑结构的分句方式将候选边界数量减少约一半，平均每条轨迹从 397 个降至 187 个。

Stage 2: Boundary scoring. An LLM judge (GPT-5.x in our case) evaluates each inter-sentence boundary as a potential breakpoint, scoring from 0 (mid-thought, would disrupt flow) to 3 (major transition, natural chapter boundary). The prompt instructs: “Never score 2–3 mid-calculation. Score 0 if previous sentence ends with ‘:’ or ‘=’.” Because traces contain hundreds of boundaries, we score them in batches: the judge sees a window of consecutive sentences and scores each boundary within that window. See Figure 1 for an example of boundary scoring.

阶段 2：边界评分。 LLM 评审器（本文使用 GPT-5.x）将每个句间边界视为潜在切分点并打分，范围为 0 分（处于思考中途，切分会破坏连贯性）至 3 分（重大转折，是自然的章节边界）。提示词要求：“绝不要在计算中途给出 2–3 分。如果前一句以‘:’或‘=’结尾，则给 0 分。” 由于每条轨迹包含数百个边界，我们采用批量评分：评审器看到一个由连续句子组成的窗口，并对窗口内的每个边界打分。边界评分示例见图1。

Memento 图1：boundary-scoring — 图1：边界评分为每个句间边界赋予 0–3 分。第 179 句完成一个结论（2.0 分）；第 180 句转向新策略（3.0 分，为最强的边界）；第 185–186 句处于推导中途（0.0 分，绝不应在此切分）。分段优化器会在高分转折处选择切分点。

Stage 3: Segmentation. Given $n$ sentences with boundary scores $s_{1}, \dots, s_{n - 1}$ , we partition the trace into $K$ contiguous blocks by maximizing

阶段 3：分段。 给定 $n$ 个句子及其边界评分 $s_{1}, \dots, s_{n - 1}$ ，我们将轨迹划分为 $K$ 个连续块，方法是最大化

\frac{1}{K} \sum_{b \in boundaries} s_{b} - λ \frac{σ (ℓ_{1}, \dots, ℓ_{K})}{μ (ℓ_{1}, \dots, ℓ_{K})},

subject to every block containing at least 200 tokens. Here, $b$ is the boundary positions, $ℓ_{k}$ is the token count of the $k$ -th block, $μ$ and $σ$ are the mean and standard deviation of the block sizes, and $λ = 0.5$ . We optimize over valid partitions and values of $K$ .

并约束每个块至少包含 200 个 token。其中， $b$ 表示边界位置， $ℓ_{k}$ 为第 $k$ 个块的 token 数， $μ$ 和 $σ$ 分别为各块大小的均值与标准差，且 $λ = 0.5$ 。我们在有效划分及 $K$ 的取值上进行优化。

We observe that the first term rewards cutting at strong semantic boundaries: a partition that places cuts at score-3 transitions (major topic changes) scores higher than one cutting low-score transitions (mid-derivation). The second term penalizes uneven block sizes. The coefficient of variation $σ / μ$ is scale-invariant, so the penalty applies equally whether the trace is 5K or 50K tokens. Without this term, the optimizer would greedily cut at the top-scoring boundaries regardless of balance, often producing one very long block and several tiny ones.

可以看到，第一项鼓励在明显的语义边界处切分：在 3 分转折处（重大主题变化）切分的方案，得分高于在低分转折处（推导中途）切分的方案。第二项惩罚块大小的不均衡。变异系数 $σ / μ$ 与尺度无关，因此无论轨迹包含 5K 还是 50K 个 token，这一惩罚都同样适用。没有这一项时，优化器会不顾均衡性地贪心选择得分最高的边界，常常产生一个很长的块和若干很短的块。

Stage 4: Iterative memento generation. Each block is compressed into a memento: a terse state representation that preserves all logically relevant information (definitions, formulas, intermediate values, chosen strategies, rejected approaches) needed for subsequent blocks to succeed. Unlike traditional summarization, the goal is “lossless compression” of reasoning state: mementos must capture everything a future reasoning step might need, targeting ∼15–25% of original tokens while being purely extractive (no new derivations or error corrections).

阶段 4：迭代生成 memento。 每个块被压缩为一个 memento：一种精简的状态表示，保留后续块成功推理所需的所有逻辑相关信息，包括定义、公式、中间数值、所选策略以及被否定的方法。与传统摘要不同，目标是 对推理状态进行“无损压缩”：memento 必须包含未来推理步骤可能需要的所有内容，长度目标约为原始 token 数的 15%–25%，同时严格采用抽取方式，不进行新的推导或错误修正。

Compressor. The compressor call (using GPT-5.x) receives all blocks and produces one memento per block using terse notation (semicolon-separated clauses, “name: value” pairs, compact math). The prompt instructs: “You are a STATE-COMPRESSOR. Minimize tokens subject to fully capturing all logically relevant information.”

压缩器。 压缩器调用使用 GPT-5.x，接收所有块，并以精简记法为每个块生成一个 memento，例如用分号分隔的分句、“名称：数值”对和紧凑的数学表达。提示词要求：“你是一个状态压缩器。在完整保留所有逻辑相关信息的约束下，尽量减少 token 数。”

Judge. A separate LLM call (again using GPT-5.x) evaluates each memento on a 0–10 scale across six dimensions: (1) formulas extracted verbatim (0–3), (2) numerical values preserved (0–2), (3) methods explicitly named (0–2), (4) validation included (0–1), (5) no hallucinations (0–1), and (6) result-first structure (0–1). If the score falls below the acceptance threshold $τ = 8$ (out of 10), the judge provides actionable feedback (e.g., “Missing formula: $K^{2} - 3 K + 3$ ”) used to refine the memento; mementos scoring $\geq τ$ are accepted without refinement. We use max $T = 2$ iterations, i.e., Compressor → Judge → Compressor → Judge. Adding more iterations does not significantly improve memento quality and results in longer mementos. See Figure 10 for an example of the iterative improvement of mementos.

评审器。 另一次独立的 LLM 调用（同样使用 GPT-5.x）从六个维度按 0–10 分对每个 memento 评分：（1）逐字抽取公式（0–3 分）；（2）保留数值（0–2 分）；（3）明确写出方法名称（0–2 分）；（4）包含验证（0–1 分）；（5）没有幻觉（0–1 分）；（6）采用结果优先的结构（0–1 分）。如果得分低于验收阈值 $τ = 8$ （满分 10 分），评审器就会提供可执行的反馈，例如“遗漏公式： $K^{2} - 3 K + 3$ ”，用于改进 memento；得分 ≥ $τ$ 的 memento 则直接通过，无须改进。我们最多进行 $T = 2$ 轮迭代，即压缩器 → 评审器 → 压缩器 → 评审器。继续增加迭代次数不会显著改善 memento 质量，反而会使其更长。 memento 迭代改进的示例见图10。

Iterative refinement is essential. Single-pass memento generation achieves only 28% pass rate ( $\geq 8 / 10$ judge score). Two iterations of judge feedback bring this to 92%. Initial mementos often miss critical formulas or intermediate values that downstream blocks need to reason correctly.

迭代改进至关重要。 单次 memento 生成的通过率仅为 28%，通过标准是评审得分达到 ≥ 8/10。两轮评审反馈可将其提高到 92%。初始 memento 常常遗漏下游块正确推理所需的关键公式或中间数值。

Dataset statistics. Figure 2 characterizes the final OpenMementos dataset (228K samples: 54% math, 19% code, 27% science). Math and code traces produce more blocks per sample (median 9) than science (median 7), and math has the largest blocks (median 3.8K chars). Summary sizes are remarkably stable across domains (median 509–603 chars), yielding median compression ratios of 0.16 (math), 0.18 (code), and 0.23 (science)—corresponding to ∼4–6× block-level compression. Across the full dataset, the average block contains ∼1,150 tokens and the average memento ∼194 tokens, for a trace-level compression of ∼6× (from ∼10,900 block tokens to ∼1,850 memento tokens per trace).

数据集统计。 图2 描述了最终 OpenMementos 数据集的特征：228K 个样本中，数学占 54%，代码占 19%，科学占 27%。数学和代码轨迹的每个样本包含更多块，中位数为 9，而科学为 7；数学的块也最大，中位数为 3.8K 字符。摘要长度在不同领域间十分稳定，中位数为 509–603 字符；数学、代码和科学的压缩比中位数分别为 0.16、0.18 和 0.23，相当于块大小压缩至原来的约 1/4–1/6。在整个数据集中，每个块平均包含约 1,150 个 token，每个 memento 平均包含约 194 个 token，轨迹层面压缩至原来的约 1/6，即每条轨迹从约 10,900 个块 token 降至约 1,850 个 memento token。

Memento 图2：dataset — 图2：**OpenMementos 数据集在各领域的分布**，共 228K 个样本。**(a)** 数学和代码每个样本约有 9 个块，科学约有 7 个。**(b)** 块大小从科学的 2.3K 字符到数学的 3.8K 字符不等。**(c)** 各领域的摘要长度集中在 509–603 字符，表明压缩目标较稳定。**(d)** 数学的块更大，因此压缩比最低，中位数为 0.16。

4. Training the Memento Models

We use a two-stage SFT procedure on OpenMementos that separates format learning from context management. The intuition follows standard curriculum learning: we first let the model acquire the block-memento format under normal conditions, then introduce the harder constraint of operating without access to masked content, see Section A.4.1 for an ablation.

我们在 OpenMementos 上采用两阶段 SFT，将格式学习与上下文管理分开。这一设计遵循标准课程学习的思路：先让模型在正常条件下学会块与 memento 交替的格式，再引入无法访问被屏蔽内容这一更困难的约束，消融实验见附录 A.4.1。

Stage 1: Full Attention: Standard causal attention over all tokens. Loss is computed on all tokens, including thinking blocks, mementos, special tokens, and the final answer. The model learns the block-memento format without any context management pressure.

阶段 1：完全注意力：对所有 token 采用标准因果注意力。损失在所有 token 上计算，包括思考块、memento、特殊 token 和最终答案。模型在没有上下文管理压力的情况下学习块与 memento 的格式。

Stage 2: Memento Attention: After each completed memento, the preceding thinking block is masked from all subsequent attention. This teaches the model to produce self-contained mementos that carry all information needed for downstream reasoning.

阶段 2：Memento 注意力：每个 memento 完成后，前一个思考块都会从此后的全部注意力中被屏蔽。这会教会模型生成自包含的 memento，携带下游推理所需的全部信息。

The attention mask implementation maintains a block cache that tracks whether each token belongs to a thinking block, summary, or other content. When <|summary_end|> is generated, the preceding block is marked as completed and masked from future attention. For training, this mask is constructed upfront as a dense matrix; for inference, the block cache is stateful across autoregressive steps. Four special tokens (<|block_start|>, <|block_end|>, <|summary_start|>, <|summary_end|>) are added and initialized as the mean embedding of semantically related existing tokens (e.g., <|block_start|> from block, start, begin, section, step) plus small Gaussian noise.

Data Scaling. When training “from scratch” a non-reasoning model (Qwen2.5-7B-Instruct), data scaling follows a similar monotonic trend as standard reasoning SFT. We study how performance scales with the amount of OpenMementos training data by fine-tuning Qwen2.5-7B-Instruct on varying amounts of data (1K, 3K, 10K, 31K, 100K examples), comparing vanilla OpenThoughts (OT), OpenMementos with full attention (OM/Full), and OpenMementos with memento attention (OM/Mem).

数据规模扩展。 对非推理模型 Qwen2.5-7B-Instruct 进行“从头”训练时，数据规模扩展呈现出与标准推理 SFT 类似的单调趋势。我们使用不同数量的 OpenMementos 数据（1K、3K、10K、31K、100K 个样本）微调 Qwen2.5-7B-Instruct，研究性能如何随训练数据量变化，并比较普通 OpenThoughts（OT）、采用完全注意力的 OpenMementos（OM/Full），以及采用 memento 注意力的 OpenMementos（OM/Mem）。

As shown in Figure 3, all three methods improve monotonically from 1K to 100K. OT achieves the highest accuracy across all data budgets, while OM/Full and OM/Mem trail by a modest margin.

如图3 所示，三种方法的性能都随数据量从 1K 增至 100K 而单调提升。 OT 在所有数据预算下的准确率最高，而 OM/Full 和 OM/Mem 仅略微落后。

Memento 图3：scaling — 图3：**训练数据规模扩展。** Qwen2.5-7B-Instruct 使用 1K–100K 个样本微调后，在 AIME24 和 AIME25 上的 pass@1 准确率。所有方法均随数据规模增大而单调改善。

Fine-Tuning Reasoning Models. When starting with already strong reasoning models, we found that training for more epochs on fewer samples is more effective than training on more samples with fewer epochs. We train on 31K samples from the 228K OpenMementos pool with 32K sequence length, as further gains are more effectively attainable through reinforcement learning (Section 5) rather than additional supervised data. We release the full 228K dataset to support future research in both directions.

微调推理模型。 对于本身已具有较强推理能力的模型，我们发现，用较少样本训练更多轮，比用更多样本训练较少轮更有效。我们从 228K 条 OpenMementos 数据中选取 31K 个样本，以 32K 序列长度训练，因为进一步的收益通过强化学习（第 5 节）比增加监督数据更容易获得。我们发布完整的 228K 数据集，以支持这两个方向的后续研究。

Hyperparameters. We use the same hyperparameters for all models and stages. Key hyperparameters: learning rate $8 \times 10^{- 5}$ , cosine schedule with 5% warmup, 5 epochs per stage, AdamW ( $β_{1} = 0.9$ , $β_{2} = 0.999$ ), no weight decay, bfloat16 precision, batch size 512, and 32 B200 GPUs. See Section A.2.1 for details.

超参数。 所有模型及训练阶段均采用相同的超参数。关键超参数包括：学习率 $8 \times 10^{- 5}$ ，带 5% 预热的余弦调度，每阶段 5 轮训练，AdamW（ $β_{1} = 0.9$ 、 $β_{2} = 0.999$ ），不使用权重衰减，bfloat16 精度，批大小 512，以及 32 张 B200 GPU。详细设置见附录 A.2.1。

表1：Memento 在采用统一注意力层的模型上，将峰值 KV 缓存降至原来的 1/2–1/3，同时保持较强的推理性能；在 Qwen3-8B 上进一步进行 RL 能继续提升准确率。由于采用混合滑动窗口架构，Olmo-3-7B 的节省幅度较小，约为对照的 0.85–0.93 倍（第 4 节）。Δ 列表示 Memento 和 Mem.+RL 相对 Control 的变化：准确率采用加性差值，单位为百分点；KV 采用乘性比值，即 Method / Control，因此 0.39× 表示 KV 减少 61%。指标。 Accuracy（%）：pass@1 准确率。Peak KV（GB）：KV 缓存峰值，决定服务单个请求所需的最低内存。AUC KV（GB·ktok）：KV 占用随 token 变化的曲线下面积，刻画总内存时间成本，同时惩罚较大的占用和较长的生成，示例见图4。行。对每个模型报告 Base（未经修改）、Control（使用创建 OpenMementos 的同一批 OpenThoughts 原始轨迹，以相同训练样本数微调 Base，但不添加块和 memento 标注）以及 Memento（在 OpenMementos 上使用块屏蔽进行 SFT）；对 Qwen3-8B 还报告 Memento+RL（附录 A.2.3）。Olmo-3 Memento 的竞赛数学评估每题生成 8 次，其他模型均为 64 次，并且使用 HuggingFace Transformers 而非 vLLM 运行。完整基准与评估细节见附录 A.2.2。

			AIME'26		Comp. Math		MATH-500		GPQA-D		LCB v6
			Val	Δ	Val	Δ	Val	Δ	Val	Δ	Val	Δ
Qwen3-8B	Base	Acc	66.8_1.1		54.3_0.3		90.5_0.9		61.4_2.4		73.1_1.0
		Peak KV	2.41		2.71		0.84		1.23		1.76
		AUC KV	25.3		30.9		4.3		6.6		15.6
	Control	Acc	64.7_1.1		49.2_0.3		89.7_1.0		57.8_2.5		70.0_1.0
		Peak KV	2.59		2.82		0.88		1.60		1.89
		AUC KV	28.3		33.1		4.7		11.8		19.2
	Memento	Acc	57.3_1.1	−7.4	45.1_0.3	−4.1	90.1_0.9	+0.4	55.8_2.5	−2.0	66.5_1.0	−3.5
		Peak KV	1.02	0.39×	1.08	0.38×	0.41	0.47×	0.56	0.35×	0.60	0.32×
		AUC KV	9.7	0.34×	10.7	0.32×	1.9	0.40×	4.0	0.34×	5.6	0.29×
	Mem. + RL	Acc	64.9_1.1	+0.2	49.4_0.3	+0.2	91.0_0.9	+1.3	62.9_2.4	+5.1	68.8_1.0	−1.2
		Peak KV	1.45	0.56×	1.48	0.52×	0.68	0.77×	1.24	0.77×	1.12	0.59×
		AUC KV	14.9	0.53×	16.4	0.50×	3.2	0.68×	9.2	0.78×	10.3	0.54×
Phi-4-r (14B)	Base	Acc	71.7_1.0		55.1_0.3		87.3_1.1		64.1_2.4		64.1_1.0
		Peak KV	2.65		3.06		1.43		0.80		2.45
		AUC KV	28.8		35.9		17.8		3.7		29.2
	Control	Acc	69.8_1.0		51.4_0.3		90.6_0.9		64.1_2.4		65.0_1.0
		Peak KV	3.04		3.48		1.04		2.11		2.64
		AUC KV	28.6		36.7		4.9		14.2		26.8
	Memento	Acc	67.6_1.1	−2.2	48.7_0.3	−2.7	89.7_1.0	−0.9	61.6_2.4	−2.5	61.8_1.1	−3.2
		Peak KV	1.17	0.38×	1.25	0.36×	0.51	0.49×	0.80	0.38×	0.92	0.35×
		AUC KV	11.3	0.40×	13.1	0.36×	2.6	0.53×	6.2	0.44×	9.5	0.35×
Qwen3-32B	Base	Acc	75.2_1.0		62.7_0.3		91.9_0.9		65.9_2.4		78.0_0.9
		Peak KV	3.24		3.67		1.26		1.89		2.88
		AUC KV	26.7		34.7		5.5		9.7		22.9
	Control	Acc	74.1_1.0		58.5_0.3		91.8_0.9		64.6_2.4		75.3_0.9
		Peak KV	3.83		4.51		1.36		2.45		3.05
		AUC KV	35.2		48.5		6.2		15.7		27.7
	Memento	Acc	72.6_1.0	−1.5	56.2_0.3	−2.3	91.1_0.9	−0.7	62.1_2.4	−2.5	74.0_1.0	−1.3
		Peak KV	1.67	0.44×	1.74	0.39×	0.64	0.47×	1.07	0.44×	1.12	0.37×
		AUC KV	14.0	0.40×	15.7	0.32×	2.8	0.45×	7.6	0.48×	9.3	0.34×
Olmo 3 (7B)	Base	Acc	67.9_1.1		52.7_0.3		91.3_0.9		50.8_2.5		64.5_1.0
		Peak KV	3.95		4.21		2.11		3.21		3.33
		AUC KV	50.8		60.1		10.7		30.8		40.0
	Control	Acc	59.8_1.1		48.3_0.3		90.4_0.9		45.7_2.5		58.8_1.1
		Peak KV	3.51		3.78		2.00		2.94		3.22
		AUC KV	37.1		46.0		9.1		23.8		37.6
	Memento	Acc	55.4_3.2	−4.4	48.1_0.9	−0.2	91.1_0.9	+0.7	49.5_2.5	+3.8	56.0_1.1	−2.8
		Peak KV	3.21	0.91×	3.43	0.91×	1.70	0.85×	2.72	0.93×	2.21	0.69×
		AUC KV	37.8	1.02×	43.6	0.95×	8.5	0.93×	25.2	1.06×	20.6	0.55×

Results and Evaluation. We evaluate Memento across four model families and scales: Qwen3-8B, Phi-4-reasoning (14B), Qwen3-32B, and Olmo-3-7B. Example Memento traces from Qwen3-32B are provided in Section A.5. All results report pass@1 accuracy. We evaluate on 14 benchmarks spanning competition mathematics (11 contests sourced from MathArena, grouped as “Comp. Math” in Table 1), standard math (MATH-500), science (GPQA Diamond), and code (LiveCodeBench v6); Table 1 summarizes accuracy alongside KV cache footprints.

结果与评估。 我们在四种模型家族与规模配置上评估 Memento：Qwen3-8B、Phi-4-reasoning（14B）、Qwen3-32B 和 Olmo-3-7B。 Qwen3-32B 生成的 Memento 轨迹示例见附录 A.5。所有结果均报告 pass@1 准确率。我们在 14 个基准上开展评估，覆盖竞赛数学（从 MathArena 获取的 11 项竞赛，在表1 中合并为“Comp. Math”）、标准数学（MATH-500）、科学（GPQA Diamond）和代码（LiveCodeBench v6）；表1 汇总了准确率与 KV 缓存占用。

Control Runs. To decouple the effect of performing SFT on already strong reasoning models (that leads to some performance loss) we do control (shown in Gray in Table 1) runs where we train the base models on the original unmodified OpenThoughts subsets. As one may expect the highest performance loss, both for Memento and the control runs, happens for the most challenging Competition math benchmarks while for easier benchmarks such as MATH-500 Memento is able to match baselines almost perfectly.

对照实验。 为分离对本身已较强的推理模型进行 SFT 所造成的影响（这会带来一些性能损失），我们设置了对照实验，在未经修改的原始 OpenThoughts 子集上训练基础模型，结果在表1 中以灰色展示。可以预见，无论 Memento 还是对照实验，最大的性能损失都出现在最具挑战性的竞赛数学基准上；而在 MATH-500 等较容易的基准上，Memento 几乎能完全追平基线。

Scale Helps. Within the Qwen3 family the accuracy gap shrinks with scale, from −6.3 pp at 8B to −3.5 pp at 32B (averaged across the five benchmark groups in Table 1). This suggests that larger models manage compressed context more effectively and that further gains may be achievable at greater scale.

规模增大有帮助。 在 Qwen3 家族中，准确率差距随规模增大而缩小，从 8B 时的 −6.3 个百分点降至 32B 时的 −3.5 个百分点，数值为表1 中五组基准的平均结果。这说明更大的模型能够更有效地管理压缩后的上下文，继续扩大规模可能带来进一步收益。

Peak KV cache and AUC savings. We observe that peak KV is reduced by 2–3× and KV AUC (area under the KV-cache-size curve over generation steps) capturing total memory-time cost, by 2–3.5× on competition math, with even larger reductions on benchmarks where the base model generates long responses. Figure 4 illustrates the range of per-problem KV cache behaviors produced by block masking.

峰值 KV 缓存与 AUC 的节省。 我们观察到，在竞赛数学上，峰值 KV 缓存降至原来的 1/2–1/3，反映总内存时间成本的 KV AUC（KV 缓存大小随生成步骤变化的曲线下面积）降至原来的 1/2–1/3.5；在基础模型生成较长回答的基准上，降幅更大。图4 展示了块屏蔽在不同问题上产生的多种 KV 缓存变化模式。

Memento 图4：kv-traces — 图4：**单个问题上的 KV 缓存轨迹，使用 Qwen3-8B，两个模型的答案均正确。** **(a)** AIME24 P2：包含 6 次压缩的典型锯齿模式；峰值为 0.77 GB，基础模型为 2.17 GB，降至原来的 1/2.8。**(b)** AIME24 P26：Memento 用 12K token 解题，基础模型为 23K；频繁压缩使峰值保持在 0.82 GB，基础模型为 3.41 GB，降至原来的 1/4.2。**(c)** AIME24 P5：Memento 生成约 3 倍的 token，31K 对 10K，期间进行了多次压缩。峰值仍较低，为 1.27 GB 对 1.55 GB，但 KV 总曲线下面积却增至基础模型的 2.1 倍，展示了块屏蔽诱发过度生成的一种失败模式。

Memento on Olmo-3-7B-Think. We applied our OpenMementos dataset and training recipe to Olmo-3-7B-Think, which uses a hybrid attention architecture: 24 of its 32 layers employ sliding-window attention (window size 4096), while only 8 layers use full causal attention. Additionally, Olmo 3 uses multi-head attention (MHA, 32 KV heads) rather than grouped query attention (GQA, 8 KV heads in Qwen3). Memento transferred with no architecture-specific modifications: accuracy is well preserved, with Comp. Math dropping only 0.2 pp relative to Control and MATH-500 improving by 0.7 pp (Table 1). However, KV cache savings are substantially more modest than for the other model families (∼0.85–0.93× peak vs. 0.35–0.47×).

Olmo-3-7B-Think 上的 Memento。 我们将 OpenMementos 数据集和训练方案应用于 Olmo-3-7B-Think，该模型采用混合注意力架构：32 层中的 24 层使用滑动窗口注意力，窗口大小为 4096，只有 8 层使用完全因果注意力。此外，Olmo 3 使用多头注意力（MHA，32 个 KV 头），而非分组查询注意力（GQA，Qwen3 中有 8 个 KV 头）。 Memento 无须任何针对架构的修改即可迁移：准确率保持良好，相较于 Control，Comp. Math 仅下降 0.2 个百分点，MATH-500 还提升了 0.7 个百分点（表1）。不过，其 KV 缓存节省幅度明显小于其他模型家族，峰值约为对照的 0.85–0.93 倍，而其他模型为 0.35–0.47 倍。

This is because sliding-window layers already cap their KV cache at 4096 tokens regardless of block masking—so 75% of layers gain nothing from eviction. Only the 8 full-attention layers benefit, limiting the overall reduction. On some benchmarks, the AUC metric is even slightly worse for Memento (e.g., AIME'26: 1.02×), because summaries lengthen the response, increasing total memory-time cost despite a lower peak. LCB shows the largest savings (0.69× peak, 0.55× AUC), likely because code problems have shorter responses where fewer tokens exceed the sliding window.

这是因为无论是否进行块屏蔽，滑动窗口层都已将 KV 缓存限制在 4096 个 token，因此 75% 的层无法从移除 token 中获益。只有 8 个完全注意力层受益，限制了总体降幅。在某些基准上，Memento 的 AUC 指标甚至略差，例如 AIME'26 为对照的 1.02 倍，这是因为摘要延长了回答，尽管峰值降低，总内存时间成本却增加了。 LCB 的节省幅度最大，峰值为对照的 0.69 倍、AUC 为 0.55 倍，可能是因为代码问题的回答较短，超出滑动窗口的 token 更少。

Models learn to summarize their own reasoning. After SFT on OpenMementos, models internalize the block-and-summarize process as a new capability: they produce self-contained mementos that reduce peak KV cache by 2–3× on models with uniform attention while maintaining strong accuracy across benchmarks. On MATH-500 the gap is under 1 pp; on the hardest competition math benchmarks Qwen3-32B stays within 2.6 pp on AIME'26. The gap shrinks with scale (−6.3 pp at 8B → −3.5 pp at 32B), suggesting Memento becomes increasingly effective at larger model sizes. Memento also transfers to Olmo-3-7B's hybrid sliding-window architecture with minimal accuracy loss, though KV savings are inherently limited by the sliding window's bounded cache.

模型学会概括自身推理。 在 OpenMementos 上进行 SFT 后，模型将分块与摘要过程内化为一种新能力：生成自包含的 memento，在采用统一注意力的模型上将峰值 KV 缓存降至原来的 1/2–1/3，同时在各项基准上保持较强的准确率。在 MATH-500 上，差距小于 1 个百分点；在最困难的竞赛数学基准中，Qwen3-32B 在 AIME'26 上的差距仍控制在 2.6 个百分点以内。差距随规模增大而缩小，从 8B 的 −6.3 个百分点降至 32B 的 −3.5 个百分点，表明模型越大，Memento 越有效。 Memento 也能迁移到 Olmo-3-7B 的混合滑动窗口架构，准确率损失很小，但 KV 节省幅度受到滑动窗口有限缓存的固有限制。

Compression behavior. How does compression vary across model families and benchmarks? Summary sizes are remarkably stable (median 260–615 chars), matching the training distribution (Figure 2), while block sizes vary widely across models and tasks. This confirms the model learns a consistent compression skill that generalizes to harder problems. Figure 5 shows the full CDF of compression ratios across all four Memento models, revealing that the bulk of blocks achieve 5–20× compression with a thin tail of low-compression outliers.

压缩行为。 压缩效果在不同模型家族和基准之间如何变化？摘要长度十分稳定，中位数为 260–615 字符，与训练分布（图2）相符，而块大小则随模型和任务显著变化。这证实模型学会了一种稳定的压缩技能，并能够泛化到更困难的问题。图5 展示了全部四种 Memento 模型压缩比的完整累积分布函数（CDF），可见大多数块压缩至原来的 1/5–1/20，只有少数低压缩率的异常块构成尾部。

Memento 图5：compression — 图5：**压缩比的累积分布函数**，压缩比为摘要字符数 / 块字符数。OLMo3-7B 和 Qwen3-8B 在竞赛数学上的压缩最紧凑。Phi-4 的压缩程度分布最宽，尤其是在 MATH-500 上。

Summary length is stable; compression scales with difficulty. Across four model families and four benchmarks, mementos converge to ∼260–615 characters regardless of block length, matching training targets. Compression is strongest on competition math (9–27×) and weakest on shorter-block benchmarks (6–9×), confirming the model learns a stable summary skill, not a fixed ratio.

摘要长度稳定，压缩程度随难度变化。 在四个模型家族和四项基准上，无论块长度如何，memento 都收敛到约 260–615 字符，与训练目标一致。竞赛数学上的压缩程度最高，可压缩至原来的 1/9–1/27；块较短的基准上最低，为原来的 1/6–1/9，这证实模型学会的是稳定的摘要技能，而非固定的压缩比例。

5. Improving Accuracy via RL

Capability under compression. We first investigate whether we can match the baseline performance with Memento or there is some inherent limitation due to compression. We focus on math, which is the most challenging domain for compression (Table 1). Generating $n = 64$ independent completions per problem for all three model families on AIME 2024/25/26, we find that coverage (pass@64) is nearly identical: the gap averages only 2.6 pp and the Jaccard similarity between Base and Memento solved sets averages 96.4%, reaching 100% in two of nine settings (Table 2).

压缩条件下的能力。 我们首先研究，Memento 是否能达到基线性能，还是压缩本身存在某种固有限制。我们重点关注数学，因为它是压缩最具挑战性的领域（表1）。在 AIME 2024/25/26 上，为三个模型家族的每道题生成 $n = 64$ 个独立回答后，我们发现覆盖率（pass@64）几乎相同：差距平均仅为 2.6 个百分点，Base 与 Memento 已解题集合的 Jaccard 相似度平均为 96.4%，在九种设置中的两种达到 100%（表2）。

表2：Base 与 Memento 在 AIME 上的 问题覆盖率（pass@64）和已解题集合重叠情况，每题生成

n = 64

次，每项基准包含 30 道题。

Model	Bench.	Base	Memento	Ret.	Jacc.
Qwen3-8B	AIME'24	93.3	90.0	96.4	96.4
	AIME'25	93.3	86.7	92.9	92.9
	AIME'26	86.7	90.0	100.0	96.3
Phi-4-r (14B)	AIME'24	93.3	93.3	100.0	100.0
	AIME'25	93.3	90.0	96.4	96.4
	AIME'26	93.3	90.0	96.4	96.4
Qwen3-32B	AIME'24	93.3	93.3	100.0	100.0
	AIME'25	90.0	83.3	92.6	92.6
	AIME'26	93.3	90.0	96.4	96.4

表2：Base 与 Memento 在 AIME 上的 问题覆盖率（pass@64）和已解题集合重叠情况，每题生成 $n = 64$ 次，每项基准包含 30 道题。

Majority voting recovers the gap. The coverage analysis above shows that Memento models can solve nearly the same problems as their base counterparts—they just do so less consistently. Majority voting (maj@ $k$ ) makes this concrete: as shown in the left panel of Figure 6, all three Memento SFT models match or exceed the Base pass@1 accuracy with just $k = 2$ –3 samples. For Qwen3-32B on AIME'26, maj@2 already surpasses the Base pass@1 line. This tells us two things: (1) the accuracy gap after SFT is a consistency problem, not a capability problem—the correct answers are in the distribution, they are just not the mode; and (2) RL is a natural fix, since it can sharpen the distribution toward correct traces without needing to teach new skills.

多数投票弥合差距。 上述覆盖率分析表明，Memento 模型能够解决与基础模型几乎相同的问题，只是作答一致性较弱。多数投票（maj@ $k$ ）具体体现了这一点：如图6 左图所示，三种 Memento SFT 模型都只需 $k = 2$ –3 个样本，即可达到或超过 Base 的 pass@1 准确率。对于 AIME'26 上的 Qwen3-32B，maj@2 已经超过 Base 的 pass@1 水平线。这说明两点：（1）SFT 后的准确率差距是 一致性 问题，而非能力问题，正确答案已经存在于分布中，只是不是众数；（2）RL 是一种自然的解决办法，因为它可以让分布更集中于正确轨迹，而无须教授新技能。

Memento 图6：rl — 图6：**多数投票的提升空间与 Qwen3-8B 的 CISPO RL 轨迹。** 左图：三种 Memento SFT 模型（表1）在 AIME 2026 上的 maj@ $k$ ；水平线表示各 Base 模型的 pass@1。所有模型在 $k = 2$ – $3$ 时均能达到 Base 准确率。中图：逐步 RL 训练准确率，浅色为原始轨迹，叠加 25 步移动平均。右图：每 25 步评估一次的 AIME'25 验证准确率，在第 350 步达到峰值 66.2%，表1 采用此检查点。多数投票出现并列时，在并列的最高票答案中等概率选择。

Recovering accuracy via RL. Given that the correct answers are already present in the Memento distribution, RL should improve pass@1 by reallocating probability mass toward correct compressed traces rather than by teaching entirely new skills. We fine-tune the Qwen3-8B Memento SFT checkpoint with CISPO, Clipped Importance-Sampled Policy Optimization, a GRPO variant that clips and detaches the importance-sampling weight. Similarly to MiniMax et al. (2025), we found CISPO to be more stable than standard GRPO during training. We also add a KL penalty ( $β = 0.001$ ) to prevent the response-length collapse we observed in initial runs without regularization, and adopt rule-based math rewards.

通过 RL 恢复准确率。 既然正确答案已经存在于 Memento 的分布中，RL 应当通过将概率质量重新分配给正确的压缩轨迹来提升 pass@1，而不是通过教授全新技能来实现。我们使用 CISPO（裁剪重要性采样策略优化）微调 Qwen3-8B 的 Memento SFT 检查点；CISPO 是一种 GRPO 变体，会裁剪重要性采样权重并阻断其梯度。与 MiniMax 等人（2025）的观察一致，我们发现 CISPO 在训练期间比标准 GRPO 更稳定。我们还加入了 KL 惩罚（ $β = 0.001$ ），以避免早期未加正则化实验中观察到的回答长度塌缩，并采用基于规则的数学奖励。

Rollouts use memento attention block masking via our custom vLLM engine (Section 6). Training uses sparse block-masked attention (similar to Stage 2 of SFT) to match the inference-time masking pattern. Full hyperparameters and training details are provided in Section A.2.3.

Rollout 通过我们定制的 vLLM 引擎使用 memento 注意力块屏蔽（第 6 节）。训练使用稀疏块屏蔽注意力，与 SFT 第二阶段类似，以匹配推理时的屏蔽模式。完整超参数与训练细节见附录 A.2.3。

CISPO algorithm. We use CISPO (Clipped Importance-Sampled Policy Optimization), a GRPO variant that replaces the PPO clipped surrogate objective with a stop-gradient clipped importance-sampling weight:

CISPO 算法。 我们采用 CISPO（裁剪重要性采样策略优化），这一 GRPO 变体用停止梯度的裁剪重要性采样权重，替代 PPO 的裁剪代理目标：

\begin{matrix} (1) & \begin{aligned} L = - sg ( & clip (r_{t} (θ), \\ 1 - ϵ_{low}, 1 + ϵ_{high})) \\ \cdot A_{t} \cdot \log π_{θ} (a_{t} ∣ s_{t}), \end{aligned} \end{matrix}

where $r_{t} (θ) = π_{θ} / π_{θ_{old}}$ is the importance ratio and $sg$ denotes stop-gradient. Unlike PPO clipping, which zeros out gradients when the ratio exceeds the trust region, CISPO ensures every token contributes a gradient signal—the clipped ratio acts as a fixed per-token weight. We add a KL penalty term $β \cdot D_{KL} (π_{θ} ∥ π_{ref})$ with $β = 0.001$ to prevent excessive drift from the SFT checkpoint.

其中， $r_{t} (θ) = π_{θ} / π_{θ_{old}}$ 是重要性比率， $sg$ 表示停止梯度。 PPO 裁剪会在比率超出信赖域时将梯度置零，而 CISPO 确保每个 token 都贡献梯度信号，裁剪后的比率充当固定的逐 token 权重。我们加入 KL 惩罚项 $β \cdot D_{KL} (π_{θ} ∥ π_{ref})$ ，其中 $β = 0.001$ ，以防止过度偏离 SFT 检查点。

Block length capping. Because we use accuracy as the sole reward signal, we observed the model learning to generate fewer and longer reasoning blocks, undermining the KV cache savings that block masking provides. To maintain low peak KV cache occupancy during RL rollouts, we cap individual blocks at 7K tokens: when a block exceeds this limit during generation, the vLLM engine forces a <|block_end|> token and the model continues from a new block.

限制块长度。 由于我们将准确率作为唯一奖励信号，模型学会生成数量更少、长度更长的推理块，削弱了块屏蔽带来的 KV 缓存节省。为在 RL rollout 期间维持较低的峰值 KV 缓存占用，我们将单个块限制为 7K 个 token：生成时一旦某个块超过这一上限，vLLM 引擎就会强制输出 <|block_end|> token，随后模型从新块继续生成。

The middle and right panels of Figure 6 show the training and validation trajectories. Train accuracy rises from 52.7% to 65.8% (25-step moving average) over 400 steps, while AIME'25 validation peaks at 66.2% at step 350. After RL, Memento+RL raises AIME'26 from 57.3 to 64.9 and Comp. Math from 45.1 to 49.4, while also improving GPQA-D from 55.8 to 62.9 above the 61.4 vanilla baseline. The compression remains substantial: peak KV rises from 1.08 to 1.48 GB after RL, still well below the 2.71 GB vanilla footprint. RL therefore converts the majority-voting headroom into stronger single-sample accuracy while preserving much of Memento's memory advantage.

图6 中图和右图展示了训练与验证轨迹。在 400 步训练中，训练准确率的 25 步移动平均从 52.7% 升至 65.8%，而 AIME'25 验证准确率在第 350 步达到峰值 66.2%。经过 RL 后，Memento+RL 将 AIME'26 从 57.3 提升至 64.9，将 Comp. Math 从 45.1 提升至 49.4，同时将 GPQA-D 从 55.8 提升至 62.9，超过普通基线的 61.4。压缩效果仍然显著：RL 后峰值 KV 从 1.08 GB 升至 1.48 GB，依然远低于普通模型的 2.71 GB。因此，RL 将多数投票揭示的提升空间转化为更强的单次采样准确率，同时保留了 Memento 的大部分内存优势。

Matching the baselines with RL. The pass@1 drop after SFT on OpenMementos reflects reduced consistency, not lost knowledge. Without any additional training, majority voting at $k = 3$ already recovers base-model accuracy (Figure 6, left). With RL fine-tuning we can significantly improve pass@1 accuracy and match or improve over the control run for Qwen3-8B (Table 1).

通过 RL 追平基线。 在 OpenMementos 上进行 SFT 后，pass@1 的下降反映出一致性降低，而不是知识丢失。无须任何额外训练， $k = 3$ 的多数投票就已能恢复基础模型准确率（图6 左图）。通过 RL 微调，我们可以显著提升 pass@1 准确率，并达到或超过 Qwen3-8B 对照实验的结果（表1）。

6. Inference and the Implicit KV Channel

6.1. Serving Memento Models with vLLM

Memento's block masking requires non-standard, data-dependent sparse attention: which tokens are masked depends on the generated sequence itself, not on a fixed pattern known at compile time. To the best of our knowledge, no production inference framework, including vLLM, SGLang, or TensorRT-LLM, provides a built-in mechanism for request-level custom sparse attention masks that evolve during generation. We therefore build native block masking support directly into vLLM's V1 engine, extending it so that it physically removes masked tokens from the KV cache. Our approach operates purely at the Python level of vLLM, can be installed as a simple patch on top of an existing vLLM installation, and works with the vanilla FlashAttention and FlashInfer kernels, requiring no custom sparse attention kernel. For more details on the implementation, see Section A.3.2.

Memento 的块屏蔽需要 非标准、依赖数据的稀疏注意力：哪些 token 被屏蔽取决于生成的序列本身，而非编译时已知的固定模式。据我们所知，包括 vLLM、SGLang 和 TensorRT-LLM 在内的生产级推理框架，都没有内置机制来支持在生成过程中动态变化的、请求级自定义稀疏注意力掩码。因此，我们直接在 vLLM 的 V1 引擎中构建原生块屏蔽支持，使其能够从 KV 缓存中 物理删除 被屏蔽的 token。我们的方法完全在 vLLM 的 Python 层运行，可以作为简单补丁安装到已有 vLLM 环境中，并兼容原版 FlashAttention 和 FlashInfer 内核，无须自定义稀疏注意力内核。更多实现细节见附录 A.3.2。

Memento 图7：throughput — 图7：**服务吞吐量，Qwen3-8B，单张 B200 GPU。** AIME24 重复 8 次，共 240 个请求，最大生成长度为 32K token。左图：满并发时，Memento 持续实现原来 1.75 倍的 token 吞吐量。右图：整批请求完成速度提高至原来的 1.58 倍。普通模型在 KV 缓存填满 GPU 显存后进入平台期。

Throughput experiments. We benchmark serving throughput on Qwen3-8B with AIME24 × 8 repetitions (240 concurrent requests) with 32K max tokens on a single B200 GPU. At high concurrency, vanilla vLLM becomes KV-cache-bound: throughput plateaus as the KV cache fills GPU memory. Memento's block masking frees KV cache entries as blocks complete, allowing the engine to sustain higher batch sizes and throughput throughout the run.

吞吐量实验。 我们在单张 B200 GPU 上，以 Qwen3-8B 运行 AIME24，重复 8 次，共 240 个并发请求，最大生成长度为 32K token，以评测服务吞吐量。高并发时，普通 vLLM 会受限于 KV 缓存：随着缓存填满 GPU 显存，吞吐量进入平台期。 Memento 的块屏蔽会在块完成时释放 KV 缓存条目，使引擎能在整个运行期间维持更大的批量和更高的吞吐量。

Memento sustains 4,290 tok/s vs. 2,447 for vanilla (1.75×) and completes the batch in 693s vs. 1,096s (1.58× faster). This infrastructure was also crucial for enabling RL fine-tuning with Memento: generating 32K-token training rollouts requires an inference engine that natively supports block masking during generation, since each rollout must produce and compact blocks on the fly. Without the vLLM integration, generating these long traces at the scale required for RL would be infeasible.

Memento 稳定达到 4,290 token/s，普通模型为 2,447 token/s，前者为后者的 1.75 倍；整批请求的完成时间为 693 秒，普通模型为 1,096 秒，相当于加速 1.58 倍。这一基础设施对于实现 Memento 的 RL 微调也至关重要：生成 32K token 的训练 rollout，需要推理引擎原生支持生成期间的块屏蔽，因为每次 rollout 都必须实时生成并压缩推理块。如果没有 vLLM 集成，就无法按 RL 所需的规模生成这些长轨迹。

Memento + vLLM. Memento's vLLM integration physically removes masked KV entries, sustaining 1.75× higher throughput at full concurrency on a single B200 GPU. Our vLLM implementation enabled us to perform reasoning RL by supporting on-the-fly block masking during 32K-token rollout generation.

Memento + vLLM。 Memento 与 vLLM 的集成会物理删除被屏蔽的 KV 条目，在单张 B200 GPU 满并发时，持续实现原来 1.75 倍的吞吐量。我们的 vLLM 实现支持在生成 32K token 的 rollout 时实时屏蔽推理块，从而使推理 RL 训练成为可能。

6.2. The Dual Information Stream

6.2.1. KV Cache Ablation: Do Memento KV States Carry Block Information?

Under memento attention, block content is masked for future tokens, but the memento's KV values were computed during generation while the model could still attend to the full block. Do these KV states carry useful information beyond the memento text itself? We denote thinking block $i$ as $T_{i}$ and its corresponding memento as $M_{i}$ .

在 memento 注意力中，块内容会对后续 token 屏蔽，但 memento 的 KV 值是在 生成期间 计算的，当时模型仍可关注完整的推理块。这些 KV 状态是否承载了超出 memento 文本本身的有用信息？我们将第 $i$ 个思考块记为 $T_{i}$ ，将相应的 memento 记为 $M_{i}$ 。

Experiment. We compare two inference modes on the same Qwen3-8B memento attention checkpoint:

实验。 我们在 同一个 Qwen3-8B memento 注意力检查点上比较两种推理模式：

Memento attention (normal): While generating $M_{i}$ , the model attends to all tokens in $T_{i}$ as well as the prompt and all preceding mementos. Once $M_{i}$ is complete, $T_{i}$ is masked from all subsequent attention—but $M_{i}$ 's KV cache entries, which were computed with block context, are retained. Future tokens therefore attend to memento KV states that implicitly encode block content.
Memento attention + restart: Generation of each memento proceeds in two steps. Step 1 (generation): $M_{i}$ 's text is generated identically to normal memento attention: the model attends to $T_{i}$ and produces the same summary tokens. Step 2 (KV recomputation): After $M_{i}$ is complete, we discard the KV cache and run a fresh prefill pass over the effective context: prompt + $M_{1}$ + $M_{2}$ + $\dots$ + $M_{i}$ (with standard causal masking within and across mementos). Critically, all past blocks are now masked and each memento's KV entries are recomputed attending only to the prompt and preceding mementos, not to the block it originally summarized. The generated memento text is identical in both conditions; only the KV representations differ. This isolates the question: does the information encoded in the KV states (from having attended to the block during generation) matter beyond what the memento text conveys?

Memento 注意力（正常模式）：生成 $M_{i}$ 时，模型关注 $T_{i}$ 中的全部 token，以及提示词和此前所有 memento。 $M_{i}$ 完成后， $T_{i}$ 会从此后的注意力中屏蔽，但在具有块上下文时计算出的 $M_{i}$ 的 KV 缓存条目仍被保留。因此，后续 token 关注的是隐式编码了块内容的 memento KV 状态。
Memento 注意力 + 重启：每个 memento 的生成分两步进行。步骤 1（生成）： $M_{i}$ 的文本生成过程与正常 memento 注意力完全相同：模型关注 $T_{i}$ ，产生相同的摘要 token。步骤 2（重新计算 KV）： $M_{i}$ 完成后，我们丢弃 KV 缓存，在有效上下文上重新进行一次预填充：提示词 + $M_{1}$ + $M_{2}$ + $\dots$ + $M_{i}$ ，在 memento 内部及它们之间采用标准因果掩码。关键在于，此时所有历史块都已被屏蔽，重新计算每个 memento 的 KV 条目时只能关注提示词与此前的 memento，无法关注它最初概括的推理块。两种条件下生成的 memento 文本完全相同，只有 KV 表示不同。这就单独检验了一个问题：生成期间关注推理块而编码进 KV 状态的信息，是否具有超出 memento 文本所传达内容的价值？

表3：完整 AIME24 上的 KV 消融：30 道题，Qwen3-8B memento 注意力检查点，生成长度 32K。块屏蔽准确率来自表1 的 64 次重复评估；重启实验重复 8 次。

Inference mode	Pass@1
Memento att. (normal)	66.1%
Memento att. + restart	50.8%
Δ	−15.3 pp

表3：完整 AIME24 上的 KV 消融：30 道题，Qwen3-8B memento 注意力检查点，生成长度 32K。块屏蔽准确率来自表1 的 64 次重复评估；重启实验重复 8 次。

The 15 pp drop confirms that memento KV states carry significant information from the masked blocks. Mementos function as compressed pointers into cached reasoning state, not just standalone text replacements. This distinguishes Memento from prior iterative summarization methods which discard original tokens entirely after summarization: unlike those methods, Memento retains the KV cache, and this retention is critical.

15 个百分点的下降证实，memento 的 KV 状态承载了来自被屏蔽块的大量信息。 Memento 相当于指向已缓存推理状态的压缩指针，而不仅是独立的文本替代物。这使 Memento 有别于此前在摘要生成后完全丢弃原始 token 的迭代摘要方法：Memento 会保留 KV 缓存，而且这种保留至关重要。

KV states carry reasoning capacity. Recomputing memento KVs without block access drops AIME24 accuracy from 66.1% to 50.8%. Mementos are not standalone text replacements—their cached KV representations form a high-bandwidth implicit channel that restart-based methods discard.

KV 状态承载推理能力。 在无法访问推理块的情况下重新计算 memento 的 KV，会使 AIME24 准确率从 66.1% 降至 50.8%。 Memento 并非独立的文本替代物，其缓存的 KV 表示构成了一个高带宽隐式通道，而基于重启的方法会丢弃这一通道。

Memento 图8：restart — 图8：**重启消融。** *步骤 1：* 生成 $M_{i}$ 时对 $T_{i}$ 采用完全注意力，与正常 memento 注意力相同。*步骤 2：* 丢弃 KV 缓存，仅对提示词 + $M_{1. . i}$ 进行预填充并重新计算； $T_{i}$ 已被屏蔽，因此 $M_{i}$ 的 KV 状态不再编码块信息。准确率下降 15 个百分点（表3），表明 KV 通道承载了相当重要的推理能力。

6.2.2. Probing the Implicit KV Channel

The KV ablation in Section 6.2.1 shows that memento KV states matter for downstream accuracy. But what information do they carry? We design a probing experiment that injects a known signal into a masked block and measures how much of it can be recovered from downstream memento KV states that never directly attended to that block.

第 6.2.1 节的 KV 消融表明，memento 的 KV 状态会影响下游准确率。但它们究竟承载了什么信息？我们设计了一个探针实验，向被屏蔽的块中注入已知信号，并衡量从未直接关注该块的下游 memento KV 状态能够恢复多少信息。

Experimental design. We inject a random 5-digit “passcode” (00000–99999) into the content of a target block $T_{2}$ in a real AIME'25 reasoning trace, then run a forward pass with Memento block masking (keep_last_n_blocks=0). We extract KV states (keys and values concatenated) from memento token positions at specific layers and train a probe (MLP, 512×256 hidden units, 128 bottleneck) to predict the 5 individual digits from these features. We report the average accuracy across the 5 predictions (one for each passcode digit) with the random prediction baseline being 10%. Crucially, the validation split is label-unique: no digit combination appears in both train and validation, preventing memorization.

实验设计。 我们将一个随机的五位“密码”（00000–99999）注入真实 AIME'25 推理轨迹中的目标块 $T_{2}$ ，随后使用 Memento 块屏蔽进行一次前向传播，设置为 keep_last_n_blocks=0。我们从特定层的 memento token 位置提取 KV 状态，将键和值拼接，并训练一个探针：具有 512 × 256 隐藏单元和 128 维瓶颈的 MLP，用这些特征预测密码的五个独立数字。我们报告五次预测的平均准确率，每次对应密码的一位数字，随机预测基线为 10%。关键在于，验证集划分采用 标签不重叠 的方式：任何数字组合都不会同时出现在训练集和验证集中，以防止模型记忆标签。

We evaluate three probing conditions:

我们评估三种探测条件：

Direct: Probe the KV states of memento $M_{2}$ , which can attend to the target block $T_{2}$ . This measures the upper bound of information encoded in a single memento's KV states.
Masked: Probe the KV states of memento $M_{3}$ , which cannot attend to $T_{2}$ as $T_{2}$ has already been evicted from the KV cache by the time $M_{3}$ is computed. Any signal recovered here must have propagated through the memento chain.
Causal control: Probe the KV states of memento $M_{1}$ , which precedes the target block $T_{2}$ in the sequence. Since $M_{1}$ is computed before $T_{2}$ is even generated, it cannot contain any information about the passcode. This serves as a sanity check; we expect chance-level accuracy (10%).

直接（Direct）： 探测 memento $M_{2}$ 的 KV 状态，它可以关注目标块 $T_{2}$ 。这衡量了单个 memento 的 KV 状态所编码信息的上限。
屏蔽（Masked）： 探测 memento $M_{3}$ 的 KV 状态，它无法关注 $T_{2}$ ，因为在计算 $M_{3}$ 时， $T_{2}$ 已经从 KV 缓存中移除。因此，此处恢复出的任何信号都必然经由 memento 链传播而来。
因果对照（Causal control）： 探测在序列中先于目标块 $T_{2}$ 的 memento $M_{1}$ 的 KV 状态。由于在 $T_{2}$ 生成之前就已计算 $M_{1}$ ，它不可能包含有关密码的任何信息。这是一项合理性检查，预期准确率应处于随机水平，即 10%。

We run this experiment at two scales:

我们在两种规模上开展这一实验：

Qwen3-8B: 15K samples from AIME'25 traces generated by the 8B model, with injected passcodes, probing at layers 3 and 35.
Qwen3-32B: 15K samples from AIME'25 traces generated by the 32B model, with injected passcodes, probing at layers 3 and 63.

Qwen3-8B：采用 8B 模型生成的 AIME'25 轨迹中的 15K 个样本，注入密码，并在索引为 3 和 35 的层进行探测。
Qwen3-32B：采用 32B 模型生成的 AIME'25 轨迹中的 15K 个样本，注入密码，并在索引为 3 和 63 的层进行探测。

Results. Figure 9 summarizes the findings. At the direct position, the memento text itself bears no relation to the passcode, yet the KV states recover the injected digits with 60–70% accuracy, demonstrating that KV representations encode far more information than the corresponding tokens. At the masked position, where the memento cannot attend to the target block, both models still recover the passcode well above chance (26.7% for Qwen3-8B, 23.0% for Qwen3-32B vs. 10% chance). The causal control, probing a memento that precedes the target block, shows exactly chance-level accuracy, confirming that the recovered signal is real and directional.

结果。 图9 汇总了实验发现。在直接位置上，memento 文本本身与密码无关，但 KV 状态能以 60%–70% 的准确率恢复注入的数字，表明 KV 表示编码的信息远多于对应的 token。在屏蔽位置上，memento 无法关注目标块，但两种模型仍能以明显高于随机水平的准确率恢复密码：Qwen3-8B 为 26.7%，Qwen3-32B 为 23.0%，而随机水平为 10%。因果对照探测的是先于目标块的 memento，结果恰好处于随机水平，证实恢复出的信号真实存在且具有方向性。

Table 4 further shows that leakage concentrates in deeper layers. In Qwen3-8B, an early layer (the 4th) shows near-chance masked accuracy (10.8%) while the last layer (the 36th) reaches 26.5%; the pattern repeats in Qwen3-32B (12.8% at the 4th layer vs. 22.4% at the 64th). The same trend holds for the direct condition, where deeper layers carry substantially more signal (64.9% vs. 51.6% for 8B; 68.7% vs. 53.8% for 32B). This is consistent with the residual stream accumulating information across layers.

表4 进一步表明，信息泄漏集中在较深的层。在 Qwen3-8B 中，早期层（第 4 层）的屏蔽条件准确率接近随机水平，为 10.8%，而最后一层（第 36 层）达到 26.5%；Qwen3-32B 也呈现相同模式，第 4 层为 12.8%，第 64 层为 22.4%。直接条件下也存在同样的趋势，更深层承载了明显更多的信号：8B 模型为 64.9% 对 51.6%，32B 模型为 68.7% 对 53.8%。这与残差流跨层积累信息的机制一致。

We further validate these findings with a controlled toy transformer experiment (Section A.4.2): a 4-layer model trained on synthetic data exhibits the same leakage pattern (24.9% masked accuracy vs. 10% chance), with signal decaying gradually over distance but persisting up to 7 hops from the target block. Leakage remains constant across training checkpoints even as task accuracy improves, confirming the channel is architectural—not learned.

我们进一步通过受控的简化 Transformer 实验验证了这些发现（附录 A.4.2）：在合成数据上训练的四层模型呈现出相同的泄漏模式，屏蔽条件准确率为 24.9%，而随机水平为 10%；信号随距离逐渐衰减，但在距离目标块多达 7 跳时仍然存在。即使任务准确率提高，泄漏水平在不同训练检查点之间仍保持不变，证实这一通道源于架构，而非训练习得。

KV states carry an implicit information channel. Memento KV representations propagate block information across masked boundaries—an architectural effect that complements the explicit memento text and explains why single-pass Memento outperforms restart-based methods.

KV 状态承载隐式信息通道。 Memento 的 KV 表示会跨越被屏蔽的边界传播块信息，这种架构效应与显式的 memento 文本相互补充，也解释了为什么单次生成的 Memento 优于基于重启的方法。

Memento 图9：probing — 图9：**探测隐式 KV 通道。** Qwen3-8B 和 Qwen3-32B 都能从屏蔽条件下的 memento 位置恢复密码，准确率分别为 26.7% 和 23.0%，明显高于 10% 的随机水平；因果对照则恰好处于随机水平。点线标记 10% 的随机水平，即在 10 个数字中随机猜测。

表4：更深的层承载信号。 直接与屏蔽条件（keep0）下的逐层探针准确率（%）。泄漏信号集中在更深层；早期层在屏蔽条件下的准确率接近随机水平。

	Qwen3-8B		Qwen3-32B
	Direct	Masked	Direct	Masked
4th layer (early)	51.6	10.8	53.8	12.8
Last layer	64.9	26.5	68.7	22.4
Both layers	65.2	26.7	71.2	23.0
Chance	10.0

7. Conclusion

We introduced Memento, a method that teaches language models to manage their own context by segmenting reasoning into blocks, compressing each into a dense memento, and masking completed blocks via sparse attention. Across three model families (Qwen3, Phi-4-reasoning, Olmo-3-7B-Think), Memento reduces peak KV cache by 2–3× and KV AUC by up to 3.5×, translating to 1.75× higher serving throughput, while preserving strong reasoning accuracy: Qwen3-32B loses just 2.6 pp on AIME'26 and 3.5 pp averaged across five benchmark groups. The gap shrinks with scale (6.3 pp at 8B → 3.5 pp at 32B), and our initial CISPO RL result on Qwen3-8B recovers much of the remaining single-sample gap while retaining the KV savings.

我们提出了 Memento，通过将推理划分为块、将每个块压缩成紧凑的 memento，并借助稀疏注意力屏蔽已完成的块，教会语言模型管理自身上下文。在三个模型家族（Qwen3、Phi-4-reasoning、Olmo-3-7B-Think）上，Memento 将峰值 KV 缓存降至原来的 1/2–1/3，KV AUC 最多降至原来的 1/3.5，使服务吞吐量达到原来的 1.75 倍，同时保持较强的推理准确率：Qwen3-32B 在 AIME'26 上仅下降 2.6 个百分点，在五组基准上的平均下降为 3.5 个百分点。这一差距随规模增大而缩小，从 8B 的 6.3 个百分点降至 32B 的 3.5 个百分点；在 Qwen3-8B 上的初步 CISPO RL 实验恢复了大部分剩余的单次采样准确率差距，同时保留了 KV 节省效果。

A key finding is that mementos carry information from masked blocks through two complementary channels: the explicit summary text and the implicit KV representations computed while the block was still visible. Our KV ablation shows that removing this implicit channel degrades accuracy by 15 pp, distinguishing Memento from methods that simply discard context after summarization.

一项关键发现是，memento 通过两个互补通道携带被屏蔽块的信息：显式摘要文本，以及在推理块仍可见时计算得到的隐式 KV 表示。 KV 消融实验表明，移除这一隐式通道会使准确率下降 15 个百分点，这使 Memento 有别于那些在生成摘要后直接丢弃上下文的方法。

Looking forward, we see two natural extensions: scaling the RL recipe to larger models, and applying Memento to long-horizon agent tasks where agent steps form natural blocks and context windows are the primary bottleneck. We release OpenMementos (228K annotated reasoning traces) and our vLLM fork with native block masking support to facilitate further research.

展望未来，我们认为有两个自然的扩展方向：将 RL 训练方案扩展到更大的模型，以及把 Memento 应用于长程智能体任务，这类任务中的智能体步骤天然构成块，而上下文窗口是主要瓶颈。我们发布了 OpenMementos（228K 条带标注的推理轨迹）和支持原生块屏蔽的 vLLM 定制版本，以推动后续研究。

🤖 Rasa

MEMENTO: Teaching LLMs to Manage Their Own Context ​

MEMENTO：教会大语言模型管理自身上下文 ​

Abstract ​

1. Introduction ​

2. Related Work ​

3. OpenMementos Dataset ​

4. Training the Memento Models ​

5. Improving Accuracy via RL ​

6. Inference and the Implicit KV Channel ​

6.1. Serving Memento Models with vLLM ​

6.2. The Dual Information Stream ​

6.2.1. KV Cache Ablation: Do Memento KV States Carry Block Information? ​

6.2.2. Probing the Implicit KV Channel ​

7. Conclusion ​