Memory Tax: Dissecting the Net Value of Agent Memory
MemoryAgentEvaluationUIUC清华大学中国人民大学Verdent.ai滑铁卢大学https://arxiv.org/abs/2610.34073v1
https://hexuan21.github.io/MemoryTax/
记忆税：剖析智能体记忆的净价值
Abstract
Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history.
记忆越来越广泛地用于长时程智能体，但相比直接从原始历史中检索，构建记忆的净价值仍在很大程度上尚不明确。 通过在多个基准和模型骨干上开展大规模评测，我们发现许多专门的记忆系统在不少情形下不及朴素 RAG，表明其净价值为负。 以 RAG 为参照，我们将记忆的净价值分解为相较直接检索的收益与损失，分别称为记忆收益和 记忆税。 这些收益和损失之所以出现，是因为记忆处理改变了原始信息的组织与访问方式。 我们将这些变化视为对原始历史的偏离，而这种偏离具有双面性。
They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community.
这些偏离最初是为帮助完成任务而设计和引入的，例如重新组织事实或提炼偏好，但其中一些也可能移除或遮蔽证据，从而产生记忆税。 由于记忆收益体现的是设计所预期的增益，我们随后更聚焦于被忽视的记忆税，研究它如何从系统和评测两个方面产生，并将这一概念推广到更多记忆应用场景。 我们希望记忆税这一概念能提供新的视角，促使人们将更多注意力从记忆系统自身的性能转向其净价值，为更可靠的记忆设计与评测提供依据。 我们将向社区开源代码和结果。
1. Introduction
Memory has become an important component of long-horizon LLM agents, supporting recall across interaction histories and reuse of prior experience. Recent work has introduced increasingly diverse memory representations and management mechanisms, aiming to make relevant information easier to preserve and utilize, but often incur additional computation over the original context. As memory systems grow more complex, it remains unclear how much these operations improve over a straightforward alternative, such as directly retrieve from the raw history.
记忆已成为长时程 LLM 智能体的重要组成部分，支持对交互历史的回忆和对既往经验的复用。 近期工作引入了日益多样的记忆表示和管理机制，旨在让相关信息更易于保存和利用，但相较直接使用原始上下文，这些机制往往需要额外的计算。 随着记忆系统越来越复杂，这些操作究竟比直接从原始历史中检索等简单替代方案改善了多少，仍不清楚。
Surveying prior works, it shows many memory designs primarily compare against other memories, while naive RAG is often overlooked, and when included, its setup is also not always fully specified (Appendix B.2). Without a retrieval baseline, the value of building memory remains entangled with the value of direct evidence retrieval, leaving its marginal or net value underexamined. So this suggests a gap between the growing complexity of memory design and our understanding of whether and where that complexity actually helps, motivating us to revisit a fundamental question:
回顾既有工作可以发现，许多记忆设计主要与其他记忆系统比较，往往忽略朴素 RAG；即使将其纳入比较，也并非总会完整说明其配置（附录 B.2）。 没有检索基线，构建记忆的价值就会与直接检索证据的价值 纠缠在一起，其边际价值或 净价值 因而未得到充分考察。 这表明，记忆设计日益增长的复杂性，与我们对这种复杂性是否以及在何处真正有用的认识之间存在缺口，促使我们重新审视一个根本问题：
What is the net value of memory over a straightforward baseline like naive RAG?
相较朴素 RAG 这样的简单基线，记忆的净价值是什么？
To study this question, we start from a common setting, personal-agent conversational memory. We evaluate representative memory designs across several benchmarks and model backbones, using naive RAG as a reference to assess the net value of memory. The results show that naive RAG remains a strong baseline and many memory systems underperform it across diverse settings, indicating a negative net value in these cases. But a closer query-level comparison shows memory is not uniformly worse.
为研究这一问题，我们从个人智能体的对话记忆这一常见场景入手。 我们在多个基准和模型骨干上评测具有代表性的记忆设计，以朴素 RAG 为参照评估记忆的净价值。 结果表明，朴素 RAG 仍是一个强基线，许多记忆系统在多种设置下都不及它，说明这些情形下的 净价值为负。 不过，进一步在查询层面进行比较发现，记忆 并非全面更差。
It does solve some queries RAG misses, where organization or restructuring may help, while losing others that RAG already solves. We refer to the former as memory benefit and the latter as memory tax. When the tax outweighs the benefit, memory falls below RAG overall.
它确实能解决一些 RAG 未能解决的查询，在这些查询中，组织或重构信息可能有所帮助；与此同时，它也会在 RAG 已能解决的其他查询上失利。 我们将前者称为记忆收益，将后者称为 记忆税。 当税负超过收益时，记忆系统的整体表现就会低于 RAG。
These gains and losses suggest a different view of memory. Memory goes far beyond a purely beneficial augmentation as often treated in prior works, and can be better understood as a double-edged process that changes how information from raw history is structured and accessed. We view such changes as inevitable deviations from the raw history. These deviations give memory its dual nature. They are originally designed and intended to better serve the task, e.g., by organizing scattered facts, distilling preferences, etc., yet some may also have unintended harmful effects due to particular design choices, such as losing or obscuring evidence. Memory benefit and tax thus capture the two sides, and their balance defines the net value of memory, reflected in the external performance.
这些收益和损失提示我们从另一种角度理解记忆。 记忆远非既有工作常常假定的那种只有好处的增强手段；更恰当的理解是，它是一种具有双面性的过程，会改变原始历史中信息的结构和访问方式。 我们将这些变化视为对原始历史的 不可避免的偏离。 这些偏离赋予了记忆 双重性质。 它们最初是为更好地服务任务而设计的，例如组织分散的事实、提炼偏好等；然而，某些偏离也可能因具体设计选择而产生意料之外的负面影响，例如丢失或遮蔽证据。 因此，记忆收益和记忆税刻画了这两个方面，二者的平衡定义了记忆的净价值，并体现于外部性能。
Memory benefit reflects the intended gains of memory design, such as reorganization or abstraction, which is expected. So we focus more on how the memory tax emerges. We first examine faithfulness for how well the information survives memory process, and then accessibility for how easily the preserved information can be recalled or accessed. We further study robustness for reliability of memory across backbone changes. These dimensions characterize memory tax from system side, and its balance with memory benefit also highly depends on the task setting. Across more personal-agent memory benchmarks, we find naive RAG already achieves high absolute scores, suggesting a retrieval-friendly task setting with limited benchmark headroom for memory to deliver more benefit and thus making the tax more visible.
记忆收益体现了记忆设计预期带来的增益，例如重组或抽象，这些增益本就在预期之中。 因此，我们更关注记忆税如何产生。 我们首先考察 忠实性，即信息在记忆处理后保留得有多好；随后考察 可访问性，即保留下来的信息有多容易被回忆或访问。 我们进一步研究 鲁棒性，即记忆在骨干模型变化时的可靠性。 这些维度从系统侧刻画记忆税，而它与记忆收益之间的平衡也高度依赖任务设置。 在更多个人智能体记忆基准上，我们发现朴素 RAG 已取得较高的绝对分数，这说明这些任务设置有利于检索，留给记忆创造更多收益的 基准提升空间 有限，也使记忆税更加显眼。
Moreover, the benefit-tax view is generalizable and not tied to particular memory forms, as nontrivial memory operations can introduce both beneficial and harmful deviations from the source. We further examine this in experiential memory and world-model memory. Finally, we derive some design implications from the perspective of memory deviations and benefit-tax trade-offs. We focus on simple changes that can be applied to existing memories without introducing new architectures.
此外，收益—税负视角可以推广，并不局限于特定记忆形式，因为任何非平凡的记忆操作都可能引入相较源信息有益或有害的偏离。 我们在 经验记忆 和 世界模型记忆 中进一步考察这一点。 最后，我们从记忆偏离以及收益与税负权衡的角度，归纳若干设计启示。 我们聚焦于无需引入新架构、即可应用于现有记忆系统的简单改动。
We hope that the notion of memory tax can shift more attention from standalone memory performance to the net value of memory and its dissection into benefit and tax, thus informing more reliable memory design and evaluation. Our contributions are as follows:
我们希望记忆税这一概念能够 促使人们将更多注意力从记忆系统自身的性能转向记忆的净价值，以及对收益与税负的分解，从而为更可靠的记忆设计与评测提供依据。 我们的贡献如下：
- Reassess the net value of long-term memory over direct retrieval holistically.
- Dissect the net value into memory benefit and memory tax, and systematically analyze the source of memory tax.
- Generalize the benefit-tax view to broader settings and derive general design implications.
- 全面重新评估长期记忆相较直接检索的净价值。
- 将净价值分解为记忆收益和记忆税，并系统分析记忆税的来源。
- 将收益—税负视角推广到更广泛的场景，并归纳一般性的设计启示。
2. Preliminary
Long-term memory for agents maintains information from past interactions and selectively exposes it to the model at query time. Let the interaction history be
智能体的长期记忆保存过去交互中的信息，并在查询时有选择地将其呈现给模型。 设交互历史为
Naive chunk-based RAG provides a natural reference under this formulation. It keeps raw history with minimal processing and retrieves relevant chunks directly. We use it throughout this work as the comparison reference for assessing the net value of memory, and measured memory benefit and memory tax are thus relative to this reference.
在这一形式化描述下，基于分块的朴素 RAG 提供了一个自然的参照。 它仅对原始历史做最少的处理并予以保留，直接检索相关文本块。 本文始终以其为比较参照，评估记忆的净价值，因此测得的记忆收益和记忆税也都是相对于这一参照而言的。
3. Experiments
3.1. Setup
Benchmarks. We evaluate on LongMemEval-s, PersonaMem-v2, and MINT-Eval. Due to cost constraints, we evaluate 20% of PersonaMem-v2 and 10% of MINT-Eval. All subsets are selected with stratified sampling over question types. These benchmarks span direct recall to more complex queries involving aggregation or multi-interference reasoning. Appendix A.1 shows the full details.
基准。 我们在 LongMemEval-s、PersonaMem-v2 和 MINT-Eval 上进行评测。 受成本限制，我们评测了 PersonaMem-v2 的 20% 和 MINT-Eval 的 10%。 所有子集均按问题类型进行分层采样。 这些基准涵盖从直接回忆，到涉及聚合或多重干扰推理的更复杂查询。 完整细节见 附录 A.1。
Baselines. We group the memory systems by how they organize what they store, and take open-source representatives of each group.
基线。 我们按照记忆系统组织所存储内容的方式进行分组，并从每组中选取开源的代表系统。
- Flat store: A-Mem, SimpleMem, MemOS.
- Hier/Lifecycle: MemoryOS, LightMem, OpenViking.
- Graph-based: HippoRAG2, PlugMem, MAGMA.
- Trained policy: Mem-T, Memory-R2.
- 扁平存储：A-Mem、SimpleMem、MemOS。
- 层次结构／生命周期：MemoryOS、LightMem、OpenViking。
- 图结构：HippoRAG2、PlugMem、MAGMA。
- 经训练的策略：Mem-T、Memory-R2。
We use RAG over the raw corpus as the main memory-free baseline, with both single-round (1rd) and two-round (2rd) variants. We also include several vanilla references, including No Context, Gold Context, etc. Appendix A.2 shows the full details.
我们以在原始语料上运行的 RAG 作为主要的无记忆基线，包括单轮（1rd）和两轮（2rd）两种变体。 我们还纳入若干基础参照，包括 No Context、Gold Context 等。 完整细节见 附录 A.2。
Implementation details. We run every system on three backbones, Qwen3.5-4B, Qwen3.6-27B and GPT-5.6-luna(low), sharing the same embedding model, Qwen3-Embedding-0.6B. Answers are scored by an LLM-as-Judge backed by Qwen3.6-27B, with judge prompts in Appendix F.2. Alongside accuracy we report several cost metrics including token cost, end-to-end wall-clock time, number of LLM calls, and memory storage size.
实现细节。 我们在三个骨干模型上运行每个系统，分别为 Qwen3.5-4B、Qwen3.6-27B 和 GPT-5.6-luna(low)，并统一使用嵌入模型 Qwen3-Embedding-0.6B。 答案由基于 Qwen3.6-27B 的 LLM-as-Judge 评分，评判提示词见 附录 F.2。 除准确率外，我们还报告若干成本指标，包括 token 成本、端到端实际运行时间、LLM 调用次数和记忆存储规模。
Since memory granularity and storage capacity vary substantially across systems, a fixed top-k can correspond to very different amounts of retrieved information. For a fair comparison, we align retrieval budgets using Retrieval Scope (RS), defined as the tokens in the memory units exposed to the LLM from retrieval through answering, normalized by the total tokens within the system's retrievable store. We tune top-k for each system (increasing it from the default in most cases) so its Retrieval Scope is roughly no lower than that of RAG (1-round), while roughly comparable across systems. The exception includes systems that contain only a few memory units, which retains a fixed top-k. See full details in Appendix A.3 and A.4.
3.2. Results
| Category | Method | LongMemEval-s | PersonaMem-v2 | MINT-Eval | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Q3.5-4B | Q3.6-27B | 5.6-luna | Q3.5-4B | Q3.6-27B | 5.6-luna | Q3.5-4B | Q3.6-27B | 5.6-luna | ||
| Vanilla | No Context | 9.0 | 9.0 | 17.6 | 20.9 | 25.6 | 28.5 | 2.3 | 3.8 | 8.5 |
| Full Context | 58.6 | 68.4 | 84.0 | 30.1 | 45.4 | 43.6 | 26.9 | 28.7 | 39.7 | |
| Gold Context | 73.4 | 84.6 | 92.8 | 58.7 | 72.9 | 69.7 | – | – | – | |
| Retrieval-only | RAG (1rd) | 70.2 | 77.6 | 82.0 | 37.2 | 46.8 | 45.1 | 25.8 | 28.6 | 29.9 |
| RAG (2rd) | 72.2 | 81.4 | 90.0 | 36.2 | 48.7 | 48.4 | 26.1 | 33.1 | 34.5 | |
| Flat store | A-Mem | 70.4(−1.8) | 79.4(−2.0) | 84.0(−6.0) | 33.7(−2.5) | 48.4(−0.3) | 41.8(−6.6) | 24.4(−1.7) | 24.4(−8.7) | 27.1(−7.4) |
| MemOS | 62.0(−10.2) | 76.6(−4.8) | 78.0(−12.0) | 37.2(+1.0) | 50.2(+1.5) | 41.7(−6.7) | 18.0(−8.1) | 19.7(−13.4) | 30.3(−4.2) | |
| SimpleMem | 60.2(−12.0) | 73.6(−7.8) | 84.0(−6.0) | 34.7(−1.5) | 46.7(−2.0) | 45.1(−3.3) | 18.6(−7.5) | 19.9(−13.3) | 22.9(−11.6) | |
| Hier/Lifecycle | MemoryOS | 64.6(−7.6) | 73.4(−8.0) | 80.0(−10.0) | 37.0(+0.9) | 48.2(−0.5) | 43.0(−5.4) | 19.0(−7.2) | 26.3(−6.9) | 33.2(−1.3) |
| LightMem | 57.6(−14.6) | 78.8(−2.6) | 77.0(−13.0) | 36.4(+0.2) | 47.0(−1.7) | 45.9(−2.5) | 20.8(−5.4) | 23.0(−10.2) | 25.2(−9.3) | |
| OpenViking | 31.8(−40.4) | 58.8(−22.6) | 63.0(−27.0) | 31.3(−4.9) | 38.7(−10.1) | 38.2(−10.2) | 10.2(−15.9) | 18.6(−14.6) | 24.2(−10.3) | |
| Graph-based | HippoRAG2 | 68.8(−3.4) | 78.2(−3.2) | 81.0(−9.0) | 33.9(−2.3) | 46.5(−2.3) | 45.3(−3.1) | 24.6(−1.5) | 23.5(−9.6) | 27.8(−6.7) |
| MAGMA | 60.8(−11.4) | 68.2(−13.2) | 76.0(−14.0) | 33.1(−3.1) | 40.5(−8.3) | 40.6(−7.8) | 23.7(−2.4) | 27.0(−6.2) | 29.5(−5.0) | |
| PlugMem | 68.0(−4.2) | 75.8(−5.6) | 70.0(−20.0) | 35.0(−1.2) | 45.2(−3.5) | 39.8(−8.6) | 21.5(−4.7) | 33.3(+0.1) | 37.8(+3.3) | |
| Trained Policy | Mem-T | 71.0(−1.2) | 75.0(−6.4) | 78.0(−12.0) | 32.1(−4.1) | 42.1(−6.6) | 41.0(−7.4) | 27.0(+0.9) | 30.4(−2.7) | 38.4(+3.9) |
| Memory-R2 | 65.8(−6.4) | 73.0(−8.4) | 79.0(−11.0) | 38.2(+2.0) | 48.4(−0.3) | 44.2(−4.2) | 21.3(−4.8) | 21.5(−11.6) | 22.9(−11.6) | |
表1：主要结果。分数为评判准确率（%）。每列的 最佳结果加粗，次佳结果加下划线。单元格按其低于或高于两轮 RAG（2rd）的幅度着色。Q3.5-4B、Q3.6-27B 和 5.6-luna 分别表示 Qwen3.5-4B、Qwen3.6-27B 和 GPT-5.6 Luna (low) 骨干模型。
图1：准确率与 token 成本的帕累托图。气泡大小表示记忆存储的 token 数。
Table 1 reports the results of all the baselines on three benchmarks, and Figure 1 plots the cost-performance trade-off more directly. Appendix E.1 to E.3 gives the full result. Given the surprisingly strong performance of naive RAG, we further validate our baseline setup in Appendix B.1 to B.3, including ablations on the RAG, reviewing RAG baselines in prior works, and validating the fairness of memory configurations. Overall, many memory systems fail to beat RAG in many cases, motivating a closer analysis in the next section.
4. Memory Tax
4.1. Overview
§3.2 shows many memory systems underperform the naive RAG despite substantially higher costs. Since RAG accesses the same raw history with minimal processing, it provides a reference for assessing the net value of building memory.
§3.2 表明，许多记忆系统尽管成本显著更高，表现却不及朴素 RAG。 由于 RAG 以最少的处理访问相同的原始历史，它为评估 构建记忆的净价值 提供了参照。
图2：逐查询比较记忆系统与单轮 RAG（1rd）回答正确情况的维恩图。
We dissect this net value through a per-query comparison with RAG (1rd) (Figure 2). Memory does solve some queries RAG misses, benefiting from additional organization or processing, while losing others that RAG already solves, where evidence may become lost, obscured, or harder to access after memory processing. These two outcomes can be understood through the inevitable deviations introduced by memory from the raw history, as any nontrivial memory process can change how information is structured, preserved, or accessed. These deviations give memory its dual nature.
我们通过与单轮 RAG（1rd）逐查询比较，剖析这一净价值（图2）。 记忆确实受益于额外的组织或处理，解决了一些 RAG 未能解决的查询；但它也会在 RAG 已能解决的其他查询上失利，因为经过记忆处理后，证据可能丢失、被遮蔽或变得更难访问。 这两种结果可以通过记忆相对于原始历史引入的 不可避免的偏离 来理解，因为任何非平凡的记忆处理都可能改变信息的结构、保存方式或访问方式。 这些偏离赋予了记忆 双重性质。
They are originally intended to better serve the tasks, yet may also have unintended harmful effects due to particular design choices, giving rise to Memory Benefit and Memory Tax. Their balance determines the net value of memory, and for the above cases, the tax outweighs the benefit. These quantities are reference-dependent, and throughout this work are defined relative to the naive RAG.
它们最初旨在更好地服务任务，但也可能因具体设计选择而产生意料之外的负面影响，从而带来 记忆收益 与 记忆税。 二者的平衡决定了记忆的 净价值；在上述情形中，税负超过了收益。 这些量取决于所选参照，本文始终将其定义为相对于朴素 RAG 的量。
Building memory has a dual nature and we call the two sides Memory Benefit and Memory Tax, with their balance defining net value of memory.
构建记忆具有双重性质，我们将其两面称为 记忆收益 与 记忆税，二者的平衡定义了记忆的净价值。
We further provide representative benefit and tax cases in Appendix D. Memory benefit often appears when extraction, aggregation, preference inference or additional processing helps as shown in several typical types in Appendix D.1, it largely aligns with the intended value of memory, while memory tax is underexplored. Its effects are visible in failed queries, but these outcomes do not explain how the losses arise within the memory. We next examine internal factors behind the memory tax.
4.2. Memory Tax - Faithfulness: How Well Is the Information Preserved?
To trace where memory tax comes from, the first place to look is how well the memory system preserves past information. Memory operations like restructuring and compression may discard or distort evidence before retrieval begins, and such loss can not be recovered later, so we first investigate the faithfulness for a memory system, at both the macro level and micro level.
要追溯记忆税的来源，首先应考察记忆系统对过去信息的保存程度。 重构、压缩等记忆操作可能在检索开始之前就丢弃或扭曲证据，而这类损失无法在之后恢复，因此我们首先从宏观和微观两个层面研究记忆系统的 忠实性。
图3：原始性能与提供金标准记忆单元后的性能（柱形），以及提供金标准记忆单元前后的记忆税（以分数量化，框内数字）。
| Method | NL | PL | ML | NU | Method | NL | PL | ML | NU |
|---|---|---|---|---|---|---|---|---|---|
| MemOS | 85.8 | 9.0 | 2.4 | 2.8 | OpenViking | 39.4 | 9.6 | 13.0 | 38.0 |
| SimpleMem | 65.6 | 27.6 | 2.2 | 4.6 | MAGMA | 93.6 | 6.2 | 0.2 | 0.0 |
| MemoryOS | 98.4 | 0.0 | 0.0 | 1.6 | PlugMem | 79.6 | 10.8 | 5.2 | 4.4 |
| LightMem | 76.2 | 13.6 | 0.6 | 9.6 | Memory-R2 | 72.6 | 14.8 | 6.0 | 6.6 |
表2：采样事实的信息损失程度标注（%）：无损失（NL）、部分损失（PL）、严重损失（ML），以及无记忆单元（NU）。
For each query, we denote the raw conversation turns or steps with the evidence as gold raw turns, and the corresponding constructed memory units as gold memory units (null if the evidence is lost after construction). At the macro level, we replay QA with only the gold raw turns and, separately, only the gold memory units. Gold memory units remove retrieval failures, so the remaining memory tax reflects the losses introduced in memory construction, as shown by the red boxes in Figure 3. At the micro level, we sample some atomic facts from the raw corpus and trace each fact to its corresponding memory units, annotating the degree of information loss, as shown in Table 2 and information loss does exist in many systems. Appendix C.2 and D.2 have more details and case studies.
Information loss with varying severity during the memory process becomes a direct cause of memory tax.
记忆处理过程中程度不一的信息损失，成为记忆税的直接来源。
However, the reduced tax after having oracle memory also indicates degradation arises beyond construction. The performance shows a similar signal. Some systems (MAGMA, LightMem) achieve gold-mem scores close to gold-raw-turn scores, yet far above their original, suggesting the preserved evidence may become harder to access.
不过，在提供理想的金标准记忆后税负有所降低，也说明性能退化并非仅源于构建环节。 性能结果也发出了类似信号。 一些系统（MAGMA、LightMem）在金标准记忆单元条件下的分数接近金标准原始轮次条件下的分数，却远高于其原始分数，这表明保留下来的证据可能变得更难访问。
4.3. Memory Tax - Accessibility: How Easily Can Stored Information Be Accessed?
§4.2 examines how much information survives memory construction. However, preserved information is useful only if it remains accessible at query time. So next we measure the accessibility of memory from both macro and micro levels.
§4.2 考察了记忆构建后有多少信息得以保留。 然而，保留的信息只有在查询时仍可访问才有用。 因此，接下来我们从宏观和微观两个层面衡量记忆的 可访问性。
图4：在原始文本块与记忆单元两种条件下，重放检索直至完整找回证据所需的轮数。
At the macro level, we replay retrieval iteratively for multiple rounds over the constructed memory and record how many rounds are needed to recover all gold memory units for a given query. Evidence lost in construction is marked as not retrievable. As shown in Figure 4, Memory systems recover fewer queries than RAG in the first round, suggesting that some relevant evidence is buried deeper after memory construction.
在宏观层面，我们对构建好的记忆 迭代重放多轮检索，记录针对给定查询找回全部金标准记忆单元所需的轮数。 构建过程中丢失的证据被标记为不可检索。 如 图4 所示，记忆系统在第一轮中完整找回证据的查询数少于 RAG，说明一些相关证据在记忆构建后被埋得更深。
图5：（查询，金标准原始轮次） 与 （查询，金标准记忆单元，橙色） 的相似度比较：左侧为整体分布偏移，右侧为逐点相似度差异。
To understand why preserved evidence becomes harder to access, we further inspect some cases as the micro level analysis (Appendix D.3). We observe several recurring patterns where evidence is fragmented across multiple memory units, diluted with unrelated information, or rephrased, weakening its accessibility compared with raw history. To visualize this effect more broadly, we compare the (query, gold raw turn) similarity with the (query, gold memory unit) similarity, reporting the point-wise changes and overall drift in Figure 5. Since each system retrieves within its own memory store, absolute similarity values across systems are only indicative, the main signal is how memory construction shifts the query alignment of the same evidence. Appendix C.3 provides more results.
Memory process can reduce evidence accessibility by fragmenting, diluting, or rephrasing it, creating memory tax even evidence is not lost.
记忆处理可能通过拆散、稀释或改写证据，降低其可访问性；即使证据没有丢失，也会产生记忆税。
4.4. Memory Tax - Robustness: How Robust Is the Memory Across Backbones?
The above analyses examine memory under a fixed backbone setting. However, long-lived agents may construct memory with one model and later consume it with another, making backbone robustness another important property. We construct memory independently with three backbones and evaluate each memory with different QA backbones.
以上分析在固定骨干模型的设置下考察记忆。 然而，长期运行的智能体可能用一个模型构建记忆，之后再由另一个模型使用，因此面对骨干模型变化的 鲁棒性 也是一项重要性质。 我们分别使用三个骨干模型独立构建记忆，并用不同的问答骨干模型评测每份记忆。
图6：每一行固定构建模型，不同标记形状表示不同的问答模型。线段表示准确率下降幅度；若降幅超过 RAG（灰色），则显示为红色，否则显示为绿色。
Two patterns emerge. 1 First, with the QA backbone fixed, changing the construction model leads to noticeable score variation. This is expected since many intermediate operations rely heavily on the backbones and can produce different memory items from the same raw history. 2 More interestingly, with the construction fixed, changing the QA backbone causes larger performance variation than naive RAG for many systems, as shown in Figure 6.
结果呈现出两种模式。 1 首先，在固定问答骨干模型时，更换构建模型会带来明显的分数变化。 这是预期之中的，因为许多中间操作高度依赖骨干模型，同一份原始历史也可能产生不同的记忆条目。 2 更有意思的是，在固定构建模型时，更换问答骨干模型会使许多系统产生比朴素 RAG 更大的性能波动，如 图6 所示。
This suggests many memory mechanisms can introduce additional fragility beyond the raw-history retrieval. In this sense, memory tax is reflected not only in point-wise performance loss, but also in the reduced robustness to backbone changes. See Appendix C.4 for more details.
这表明，相较检索原始历史，许多记忆机制可能引入 额外的脆弱性。 从这个意义上说，记忆税不仅体现在逐点的性能损失上，也体现在面对骨干模型变化时鲁棒性的下降。 更多细节见 附录 C.4。
Memory process can amplify backbone sensitivity since model-dependent operations can introduce additional variability to raw history.
记忆处理可能放大对骨干模型的敏感性，因为依赖模型的操作会在原始历史之上引入额外的变异。
4.5. Benchmark Headroom Further Exposes the Memory Tax
The analysis so far focuses on how memory tax arises within a memory system. Yet the balance between memory benefit and tax also depends on the task settings. Some tasks naturally favor direct recall from raw history, while others may benefit more from abstraction or distillation. To examine this evaluation-side effect, we run several naive baselines across a broader set of long-term memory benchmarks, including 1 No Memory, 2 IDF-keyword, a lexical matching baseline based on the keyword rarity, 3 RAG-Emb, and 4 RAG-Hybrid. Details are in Appendix C.5.
目前为止的分析聚焦于记忆税如何在记忆系统内部产生。 然而，记忆收益与税负之间的平衡也取决于任务设置。 有些任务天然适合从原始历史中直接回忆信息，而另一些任务可能更能受益于抽象或提炼。 为考察这种来自评测侧的影响，我们在更广泛的一组长期记忆基准上运行若干朴素基线，包括 1 No Memory、2 基于关键词稀有程度进行词法匹配的 IDF-keyword、3 RAG-Emb，以及 4 RAG-Hybrid。 细节见 附录 C.5。
图7：使用 Q27B 在更多记忆基准上评测检索基线的结果。细节见 附录 C.5。
Figure 7 shows that simple retrieval baselines already performs quite well on some of the benchmarks, including some widely-used ones. For example, the extremely simplified IDF-keyword based method can even achieve more than 50% in absolute score on some of them, while RAG-Hybrid reaches about 70%. Such results suggest some current evaluation settings are relatively retrieval-friendly, leaving quite limited headroom for heavy memory to improve over naive retrieval.
图7 表明，简单的检索基线在部分基准上已经表现相当出色，其中也包括一些广泛使用的基准。 例如，极度简化的 IDF-keyword 方法在其中一些基准上的绝对分数甚至超过 50%，而 RAG-Hybrid 达到约 70%。 这些结果表明，当前部分评测设置相对有利于检索，留给复杂记忆系统超越朴素检索的 提升空间十分有限。
When naive RAG already solves many queries on certain benchmark, memory has less room for additional gains and even modest losses introduced by memory processing become more apparent in performance, making the tax more visible.
当朴素 RAG 在某个基准上已能解决许多查询时，记忆获得额外收益的空间就更小，记忆处理引入的损失即使不大，也会更明显地反映在性能中，使记忆税更加显眼。
This also helps explain why some systems with lightweight or even no explicit memory construction can achieve strong performance and efficiency. When naive retrieval already provides a strong baseline, they “inherit” much of that performance while introducing little memory tax, so lightweight design can be particularly competitive under limited benchmark headroom.
这也有助于解释，为什么一些采用轻量记忆构建、甚至完全不显式构建记忆的系统，能够取得较强的性能与效率。 当朴素检索已提供一个强基线时，这些系统会 “继承”其中的大部分性能，同时仅引入少量记忆税，因此，在基准提升空间有限的情况下，轻量设计可能尤其具有竞争力。
On the other hand, many real-world queries are also naturally retrieval-friendly, making such accurate recall tasks essential and meaningful in its own right. Current benchmarks contain a large share of such queries, and future evaluations could complement them with harder settings. Moreover, reporting memory systems alongside naive RAG can make remaining improvement room more explicit, which also helps clarify the contribution of built memory. RAG scores can also serve as a reference for calibrating difficulty and leaving meaningful room for more reliable evaluations.
另一方面，现实中的许多查询也天然适合检索，因此，这类准确回忆任务本身就是必要且有意义的。 当前基准中此类查询占比很大，未来评测可以补充更具挑战性的设置。 此外，同时报告记忆系统和朴素 RAG 的结果，可以 更明确地呈现剩余的提升空间，也有助于厘清所构建记忆的贡献。 RAG 分数还可以作为 校准难度并保留有意义提升空间 的参照，从而开展更可靠的评测。
5. Generalizing Memory Tax
Memory serves a wide range of application scenarios beyond personal-agent memory. We next examine whether the same benefit-tax view extends to two substantially different settings.
除了个人智能体记忆之外，记忆还服务于广泛的应用场景。 接下来，我们考察同样的收益—税负视角能否扩展到两种差异显著的场景。
5.1. Experiential Memory
In this setting, agents accumulate experience from past task traces to guide future ones. We evaluate on several agentic scenarios covering embodied tasks (ALFWorld), web (WebShop, Mind2Web), coding (BigCodeBench) and tool use (AppWorld, τ²-Bench), with Qwen3.5-4B as backbone. Experience is accumulated on the training split and evaluated on the test split with memory frozen. We compare Memp, ReasoningBank, G-Memory and SkillPro against a ReAct agent without memory and a naive RAG baseline, which retrieves the most similar past tasks by their descriptions and places the trajectories into the context, details are in Appendix C.6.1.
在这一场景中，智能体从过去的任务轨迹中积累经验，以指导未来任务。 我们以 Qwen3.5-4B 为骨干模型，在多个智能体场景中开展评测，涵盖具身任务（ALFWorld）、网页任务（WebShop、Mind2Web）、编程（BigCodeBench）和工具使用（AppWorld、τ²-Bench）。 经验在训练集上积累，随后冻结记忆，在测试集上评测。 我们将 Memp、ReasoningBank、G-Memory 和 SkillPro，与无记忆的 ReAct 智能体以及朴素 RAG 基线比较；后者根据任务描述检索最相似的既往任务，并将其轨迹放入上下文，细节见 附录 C.6.1。
图8：经验记忆在智能体任务基准上的结果。记忆基线高于 RAG 时为绿色，否则为红色。包含记忆税部分的完整图见 附录 C.6.1。
Figure 8 shows the effect of memory varies across task domains. In web-based tasks, distilling long traces into workflows or strategies can outperform naive RAG, while for tool-use, several memory methods fall behind it, suggesting memory tax also emerges in experiential memory, but its magnitude depends strongly on the task settings.
图8 表明，记忆的作用因任务领域而异。 在网页任务中，将长轨迹提炼为工作流或策略可以优于朴素 RAG；而在工具使用任务中，若干记忆方法却落后于它。这说明 经验记忆中同样会出现记忆税，但其大小高度依赖任务设置。
5.2. World-Model Memory
Long-horizon world models need to retain information about past environment states to maintain temporal and scene consistency. Existing approaches have explored diverse memory representations and mechanisms, including raw frames, visual latents, and KV caches. We evaluate representative methods on two settings for long-horizon consistency: Context-as-Memory and MemoBench. Full details are in Appendix C.6.2.
长时程世界模型需要保留过去环境状态的信息，以维持时间与场景的一致性。 现有方法探索了多种记忆表示和机制，包括原始帧、视觉潜变量和 KV 缓存。 我们在 Context-as-Memory 和 MemoBench 两种长时程一致性设置下，评测具有代表性的方法。 完整细节见 附录 C.6.2。
| Method | Mem. Form | C-as-M-eval | MemoBench | ||
|---|---|---|---|---|---|
| SSIM ↑ | 1 − LPIPS ↑ | SSIM ↑ | 1 − LPIPS ↑ | ||
| I3DM | naive frames | .560 | .704 | .430 | .476 |
| WorldKV | raw KV | .372 | .459 | .379 | .420 |
| ReWorld | learned KV | .378 | .510 | .387 | .422 |
| MagicWorld | visual latents | .388 | .341 | .433 | .412 |
| FramePack | visual latents | .217 | .312 | .279 | .303 |
| HyDRA | mem. tokens | .320 | .319 | .355 | .326 |
表3：采用不同记忆表示与机制的世界模型的长时程一致性。
图9：三个评测场景的示例帧序列，用于测试长时程场景一致性。
A similar pattern appears in these settings (Table 3). Directly retrieving and reinjecting past observations achieves strong consistency, while latent- or cache-based memories do not consistently outperform this simpler alternative. This suggests that memory tax can also arise in world-model memory, where additional memory operations may introduce losses that offset their intended benefits. It also motivates exploration of more unified memory carriers in video world models.
这些设置下也出现了类似模式（表3）。 直接检索并重新注入过去的观察结果，能够取得较强的一致性；而基于潜变量或缓存的记忆并未稳定地优于这一更简单的替代方案。 这表明 世界模型记忆中同样可能产生记忆税，其中额外的记忆操作可能引入损失，抵消其预期收益。 这也促使我们探索视频世界模型中更统一的记忆载体。
Memory process in a broader sense can also introduce both beneficial and harmful deviations from the original information, giving rise to memory benefit and tax, making the concepts generalizable across memory forms and scenarios.
更广义的记忆处理同样可能引入相对于原始信息有益和有害的偏离，带来记忆收益与记忆税，使这些概念能够推广到不同的记忆形式和场景。
Moreover, we also expect memory benefit/tax to extend beyond above settings along other axes, like omni-modality, latents, parameters, etc. We leave the exploration for future work.
此外，我们还预期记忆收益／税负能够沿其他维度扩展到上述场景之外，例如 全模态、潜变量、参数 等。 我们将这些探索留作未来工作。
6. Design Implications
In this section, we derive some simple yet effective design implications for reducing or compensating for the memory tax. We focus on improvements that can be used on existing memory systems without introducing another new architecture, and deliberately exclude common test-time scaling techniques like retrieval depth / width expansion, query refinement, using stronger backbones, etc.
本节归纳若干简单而有效的设计启示，用于 减少或补偿 记忆税。 我们聚焦于无需再引入新架构、即可用于现有记忆系统的改进，并刻意 排除常见的测试时扩展技术，例如增加检索深度或宽度、优化查询、使用更强的骨干模型等。
Qwen3.6-27B 时，应用设计启示后的准确率与记忆税（%）。| Method | LongMemEval-s | PersonaMem-v2 | ||
|---|---|---|---|---|
| Acc (↑) | Tax (↓) | Acc (↑) | Tax (↓) | |
| SimpleMem | 73.6 | 12.0 | 46.7 | 11.2 |
| + src-aug | 82.8 | 5.2 | 49.6 | 7.2 |
| + recursive | N/A | N/A | 46.8 | 12.1 |
| MemoryOS | 73.4 | 11.4 | 48.2 | 17.1 |
| + src-aug | 76.4 | 7.8 | 53.0 | 8.5 |
| + recursive | N/A | N/A | 49.1 | 16.4 |
| Mem-T | 75.0 | 10.8 | 42.1 | 15.2 |
| + src-aug | 80.0 | 5.2 | 49.0 | 8.7 |
| + recursive | N/A | N/A | 45.7 | 13.6 |
表4：使用 Qwen3.6-27B 时，应用设计启示后的准确率与记忆税（%）。
6.1 (Memory Construction) Make memory deviations more recoverable.
6.1 （记忆构建） 让记忆偏离更易于恢复。
As we discussed above, building memory can introduce changes and deviations from the raw history about how information is preserved and accessed through diverse memory operations. While no single design can predict and fit all tasks needs, and the harmful deviations are difficult to eliminate entirely, a more promising way to reduce memory tax is to make such deviations recoverable, preserving a lightweight path back to the source history when memory becomes insufficient. As a simple proof of concept, we retrieve from source history alongside normal memory retrieval and jointly rerank them before QA. This lightweight recovery path can restore evidence lost during construction or missed in access. Results in Table 4 show consistent improvements and reduction of memory tax across several memory systems.
如前所述，构建记忆会通过多种记忆操作，在信息的保存和访问方式上引入相对于原始历史的变化与偏离。 由于没有一种设计能够预测并满足所有任务需求，有害偏离也难以彻底消除，因此，降低记忆税的一种更有希望的方式，是让这些偏离 可恢复：当记忆不足时，保留一条返回原始历史的轻量路径。 作为简单的概念验证，我们在正常检索记忆的同时检索原始历史，并在问答之前对二者联合重排序。 这条轻量的恢复路径能够找回构建时丢失或访问时遗漏的证据。 表4 的结果表明，这一方法在多个记忆系统中均带来了稳定的改善，并降低了记忆税。
6.2 (Memory Access) Recursive or self-referential memory for retrieval.
6.2 （记忆访问） 面向检索的递归或自指记忆。
Memory tax can only be visible in actual use, when relevant evidences become harder to access and are recovered only after deep search. Such retrieval experience directly reflects the harmful deviations memory introduced, and can in turn provide feedback to improve future memory access, creating a recursive loop between memory access and construction. Inspired by progress on recursive self-improvement (RSI), where a system is expected to “improve how it improves”, memory can similarly “remember how it remembers”. Past queries, evidences, and retrieval paths can themselves be source materials for memory, and this is particularly useful for multi-round retrieval, since evidence found in deeper rounds can then be accessed directly by similar information needs.
记忆税只有在实际使用中才会显现：相关证据变得更难访问，只有经过深入搜索才能找回。 这种检索经验直接反映了记忆引入的有害偏离，又能够反过来提供反馈，改善未来的记忆访问，在记忆访问与构建之间形成 递归 循环。 递归自我改进（RSI）的进展启发了这一想法：RSI 期望系统 “改进其改进方式”，类似地，记忆也可以 “记住自己如何记忆”。 过去的查询、证据和检索路径 本身就可以成为记忆的源材料；这对多轮检索尤其有用，因为后续遇到相似的信息需求时，就可以直接访问原先在更深轮次中找到的证据。
It also offers a new dimension of test-time scaling at query-side, allowing inter-query scaling beyond a single query. As a simple proof of concept, we treat past queries and retrieved evidence as additional inputs into memory store, making them available for future retrieval. Results in Table 4 also validate its effectiveness.
它还在查询侧提供了测试时扩展的新维度，使扩展能够超越单个查询，发生在多个查询之间。 作为简单的概念验证，我们将过去的查询及检索到的证据作为额外输入写入记忆存储，使其可供未来检索。 表4 的结果也验证了其有效性。
6.3 (Future Direction and Beyond Memory Operations) Try modeling future demand beyond past information.
6.3 （未来方向及记忆操作之外） 尝试在过去信息之外，对未来需求进行建模。
Many methods build memory ahead of time (AoT) before future queries come. As we discussed before, memory makes deviations on the raw history, with the intention of better serving task queries, but may also add implicitly harmful effects. Without knowing future demand accurately, it is unclear what kind of deviations will be beneficial. To cover a broader range of possible queries, AoT systems may need to process more information in more elaborate ways, making it increasingly expensive. The opposite is just-in-time (JiT) memory, which delays memory operations until queries arrive. With query already known, JiT can choose more targeted forms and directions of deviation to maximize the memory benefit and minimize the memory tax, but moves computation to the online part and increases user-facing latency, to which the users are highly sensitive.
许多方法在未来查询到来之前 提前（ahead of time，AoT） 构建记忆。 如前所述，记忆对原始历史引入偏离，本意是更好地服务任务查询，但也可能带来潜在的有害影响。 如果不能准确了解未来需求，就无法明确哪种偏离会有益。 为覆盖更广泛的潜在查询，AoT 系统可能需要以更复杂的方式处理更多信息，使成本不断上升。 与之相对的是 即时（just-in-time，JiT） 记忆，它将记忆操作推迟到查询到来之后。 由于此时查询已知，JiT 能够更有针对性地选择偏离的形式与方向，以最大化记忆收益、最小化记忆税；但这会将计算移到在线阶段，增加用户直接感受到的延迟，而用户对这类延迟非常敏感。
A middle ground is to model the future demand. In personal agent settings, task queries are not purely random. Past interests, habits, and access patterns provide a prior over what information is likely to matter, and hence what kinds of deviations that memory introduces are likely useful. Such signals may help construction allocate effort more selectively, and reduces harmful deviations, without paying the full online cost of JiT. From this view, memory can partly be seen as an implicit recommendation system that estimates future relevance of past information. Exploring this perspective in real-world personal-agent settings may be a promising direction for future exploration.
一种折中方案是 对未来需求进行建模。 在个人智能体场景中，任务查询并非完全随机。 过去的兴趣、习惯和访问模式，为哪些信息可能重要提供了先验，也由此提示记忆引入的哪些偏离可能有用。 这些信号可以帮助构建过程更有选择地分配工作量、减少有害偏离，同时无需承担 JiT 的全部在线成本。 从这一角度看，记忆在一定程度上可以视为一种 隐式推荐系统，用于估计过去信息在未来的相关性。 在真实的个人智能体场景中探索这一视角，可能是未来一个有前景的方向。
7. Related Works
Long-Term Agent Memory. Long-term memory has become an important component for long-horizon agents, supporting the retention of specific information, past interactions, and task experience. Existing systems explore a variety of memory representations, including relatively flat stores, hierarchical organizations, from short-/long-term memory layers to filesystem-tree designs in recent industrial systems, and graph-based representations. Another line of variation lies in how memory operations are governed. Some systems rely on predefined harness for writing, updating, and using memories, while others allow an LLM agent to decide these operations dynamically or explore learned memory policies. In this work, we study these memory systems through the lens of their net value, examining the gains and the losses introduced by memory operations.
智能体长期记忆。 长期记忆已成为长时程智能体的重要组成部分，支持保留特定信息、过去交互和任务经验。 现有系统探索了多种记忆表示，包括相对扁平的存储、层次化组织，以及图结构表示；其中层次化组织既包括短期／长期记忆分层，也包括近期工业系统中的文件系统树设计。 另一个差异维度是记忆操作的控制方式。 一些系统依靠预定义的控制框架来写入、更新和使用记忆，另一些则允许 LLM 智能体动态决定这些操作，或探索学习得到的记忆策略。 本文从 净价值 的视角研究这些记忆系统，考察记忆操作带来的收益和损失。
Empirical Studies of Agent Memory. Recent empirical studies examine agent memory from different perspectives, including how memory systems are designed and deployed, their performance-cost trade-offs, and where computation and time are spent. These works mainly characterize existing memory designs and their empirical behavior, but our work focuses on the net value introduced by memory processing itself. We introduce memory tax to characterize its losses relative to direct retrieval, analyze how such losses arise from both system and evaluation sides, and derive system-agnostic design implications.
智能体记忆的实证研究。 近期实证研究从不同角度考察智能体记忆，包括记忆系统如何设计和部署、性能与成本的权衡，以及计算与时间花费在哪些环节。 这些工作主要刻画现有记忆设计及其经验行为，而我们的工作聚焦于记忆处理本身引入的净价值。 我们引入记忆税来刻画其相较直接检索的损失，从系统侧和评测侧分析这些损失如何产生，并归纳不依赖具体系统的设计启示。
8. Conclusion
We revisit the agent memory by studying its net value over direct retrieval from raw history. Across the evaluation on three benchmarks with many retrieval-friendly queries alongside some complex ones, many memory methods underperform naive RAG, indicating a negative net value in these settings. We dissect this net value into memory benefit and tax, showing memory can improve how past information is organized and used, but may also have harmful effects. We analyze memory tax from both system sides and evaluation settings, and generalize it to broader scenarios like experiential memory and world-model memory. We hope memory tax can shift more attention from standalone performance to the net value of memory, and encourages more reliable memory design and evaluation.
我们通过研究智能体记忆相较直接检索原始历史的净价值，重新审视智能体记忆。 在三个基准的评测中，这些基准既包含大量有利于检索的查询，也包含一些复杂查询；许多记忆方法不及朴素 RAG，说明在这些设置下其净价值为负。 我们将这一净价值分解为记忆收益和记忆税，表明记忆可以改善过去信息的组织与使用方式，但也可能产生有害影响。 我们从系统侧和评测设置两个方面分析记忆税，并将其推广到经验记忆和世界模型记忆等更广泛的场景。 我们希望记忆税能够促使人们将更多注意力从系统自身的性能转向记忆的净价值，并推动更可靠的记忆设计与评测。
Limitations
Given the rapid development of this field, our baseline selection is representative rather than exhaustive. We also follow official configurations wherever possible, though further system-specific tuning may yield better performance. Our benchmarks are primarily static, offline, and English-language, leaving long-lived deployments with interleaved queries and multilingual interactions underexplored. Extending the analysis to other forms of memory, such as omnimodal, latent, and parametric memory, is also left for future work. Our evaluation is limited by computational and API-calling cost. We do not include stronger frontier models beyond GPT-5.6 Luna (low), and use stratified subsets for some benchmarks (20% of PersonaMem-v2, 10% of MINT-Eval, and 20% LongMemEval-s questions for GPT-5.6 Luna).
鉴于这一领域发展迅速，我们选取的基线具有代表性，但并未穷尽所有方法。 我们也尽可能遵循官方配置，但进一步针对具体系统进行调优，仍可能获得更好的性能。 我们的基准主要是静态、离线且使用英语的，长期部署中查询交错到来以及多语言交互等情形尚未得到充分研究。 将分析扩展到全模态记忆、潜变量记忆和参数化记忆等其他形式，也留待未来工作。 我们的评测受到计算和 API 调用成本的限制。 我们未纳入超出 GPT-5.6 Luna (low) 的更强前沿模型，并在部分基准上使用了分层采样子集（PersonaMem-v2 的 20%、MINT-Eval 的 10%，以及为 GPT-5.6 Luna 选取的 LongMemEval-s 问题的 20%）。