MemoryTax: The Bitter Lesson of Agent Memory
MemoryAgentEvaluation上海交通大学MemoryTax：智能体记忆的苦涩教训
Abstract
Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives; coding agents keep the raw history and let the model search it with its own tools. We ask whether designed memory still helps once the model is strong enough to search for itself. On one LoCoMo conversation (419 turns, 152 questions), we run four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5, and record what every rollout costs. This is a short empirical study, with one conversation and one run per cell.
Mem0 和 Zep 等智能体记忆系统会在任何问题到来之前决定保留什么；编程智能体则保留原始历史，让模型使用自己的工具进行搜索。 我们探讨的是：一旦模型强到能够自行搜索，专门设计的记忆是否仍然有帮助。 在 LoCoMo 的一段对话（419 轮、152 个问题）上，我们运行四种记忆方法（完整上下文、BM25、Mem0、Zep）和三种编程智能体（Pi、Codex、Claude Code），分别搭配 GPT-4o-mini、GPT-6 Luna 和 Claude Opus 5.5，并记录每次完整评测的成本。 这是一项简短的实证研究，仅使用一段对话，每个实验组合只运行一次。
Agent memory is a costly patch for a weak model: Mem0 beats full context only with GPT-4o-mini, and its build alone costs more than a whole full-context rollout. Agent memory also has a ceiling, set by the model that writes it: a stronger reader does not recover what a weaker writer left out. With Claude Opus 5.5, Pi tops the cost–accuracy frontier, scoring 99% for less than half the cost of full context; its one miss is a wrong answer key. Part of that lead comes from photo captions that only the agents could read: on the 88 questions whose evidence has no photo, Pi and full context differ by one answer.
智能体记忆是弱模型的一种昂贵补丁：Mem0 仅在使用 GPT-4o-mini 时优于完整上下文，而且仅构建记忆的成本就超过了完整上下文方法的一整次评测。 智能体记忆还存在上限，这一上限由写入记忆的模型决定：更强的读取模型无法恢复较弱的写入模型遗漏的内容。 使用 Claude Opus 5.5 时，Pi 位居成本–准确率前沿的顶端，以不到完整上下文一半的成本达到 99% 的准确率；其唯一一道错题源于错误的标准答案。 这一领先优势部分来自只有智能体能够读取的照片描述：在证据不涉及照片的 88 个问题上，Pi 与完整上下文只相差一个答案。
1. Introduction
Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives: the model extracts facts or builds a graph from the conversation, and each question sees a few retrieved pieces. Coding agents and recursive language models keep the raw history instead and let the model search it with its own tools.
Mem0 和 Zep 等智能体记忆系统会在任何问题到来之前决定保留什么：模型从对话中抽取事实或构建图谱，每个问题只能看到少量检索到的片段。 编程智能体和递归语言模型则保留原始历史，让模型使用自己的工具进行搜索。
The bitter lesson is that knowledge built into a system helps at first, then plateaus, while general methods that scale with computation win. Agent memory builds in what to keep. Does it still help once the model is strong enough to search for itself?
“苦涩的教训”是指：融入系统的知识起初会有所帮助，随后便趋于停滞，而能够随计算量扩展的通用方法最终胜出。 智能体记忆将“保留什么”预先纳入了设计。 一旦模型强到能够自行搜索，这种做法是否仍然有帮助？
We run four memory methods (full context, BM25, Mem0 and Zep) and three coding agents (Pi, Codex and Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5, on a conversation from LoCoMo, and measure what each run costs. We find:
我们在 LoCoMo 的一段对话上运行四种记忆方法（完整上下文、BM25、Mem0 和 Zep）以及三种编程智能体（Pi、Codex 和 Claude Code），分别搭配 GPT-4o-mini、GPT-6 Luna 和 Claude Opus 5.5，并测量每次运行的成本。 我们发现：
- Agent memory is a costly patch for a weak model. Mem0 beats full context only with GPT-4o-mini, and its build alone costs more than full context's whole run (Section 3).
- Agent memory has a ceiling, set by the model that writes it. A stronger reader does not recover what a weaker writer left out (Section 4).
- With Claude Opus 5.5, Pi tops the cost–accuracy frontier (Figure 1). It answers every question whose answer key is sound, for less than half what full context costs. Where the two read the same text they differ by one answer; the rest of Pi's lead comes from photo captions that only the agents could read (Section 5).
The study is small: one conversation, one run per cell, and a GPT-4o-mini judge. Section 6 lists what it cannot show. Within those limits, the results suggest that a strong model needs less designed memory and better tools for searching the raw record.
这项研究的规模很小：仅有一段对话，每个实验组合只运行一次，并使用 GPT-4o-mini 作为评判模型。 第 6 节 列出了它无法证明的内容。 在这些限制之内，结果表明，强模型可能更少需要专门设计的记忆，而更需要能够搜索原始记录的优质工具。
2. Setup
We run 21 cells, seven methods with three models each, on LoCoMo conv-26: 419 turns in 19 sessions and 152 single-hop, multi-hop, temporal and open-domain questions. LoCoMo's adversarial questions are left out, as in Mem0's evaluation. The history fits every model's context window. Among the ten LoCoMo conversations, conv-26's per-method accuracies track those of the full set closely. Table 1 lists the methods.
我们在 LoCoMo 的 conv-26 对话上运行 21 个实验组合，即七种方法分别搭配三种模型：该对话包含 19 次会话中的 419 轮交流，以及 152 个单跳、多跳、时序和开放域问题。 与 Mem0 的评测一样，我们排除了 LoCoMo 中的对抗性问题。 这段历史记录能够放入每个模型的上下文窗口。 在 LoCoMo 的十段对话中，conv-26 上各方法的准确率与完整数据集上的准确率十分接近。 表1 列出了这些方法。
| Method | Stored | Model sees |
|---|---|---|
| Full context | raw history | all of it |
| BM25 | raw turns | top-8 turns, ±2 |
| Mem0 | extracted facts | top-200 facts |
| Zep | temporal graph | top-20 edges |
| Pi, Codex, Claude Code | raw history, one JSON file | what its tools read |
The memory methods answer one question at a time with Mem0's 2025 answer template; Mem0 is OSS 2.1.0 and Zep is Graphiti OSS 0.30.2, both with default extraction and retrieval. The agents, Pi 1.1.0, Codex 0.159.2 and Claude Code 2.1.295, start a fresh session in a fresh container for each question, with the conversation as one file, their default tools, and the template's answer rules in the prompt (Appendix C).
记忆方法使用 Mem0 的 2025 年回答模板，每次回答一个问题；Mem0 采用 OSS 2.1.0，Zep 采用 Graphiti OSS 0.30.2，两者均使用默认的抽取和检索设置。 这些智能体，即 Pi 1.1.0、Codex 0.159.2 和 Claude Code 2.1.295，针对每个问题都会在全新的容器中启动一个全新的会话；对话作为一个文件提供，使用各自的默认工具，并在提示词中包含该模板的回答规则（附录 C）。
The models go from weak to strongest: GPT-4o-mini, GPT-6 Luna, Claude Opus 5.5. The last two run at reasoning effort high, the effort coding agents are usually run at; GPT-4o-mini has no effort setting. In each cell one model writes the store (Mem0, Zep) and answers every question, or drives the agent.
所用模型按能力由弱到强依次为：GPT-4o-mini、GPT-6 Luna、Claude Opus 5.5。 后两者将推理力度设为 high，这也是编程智能体通常采用的设置；GPT-4o-mini 没有推理力度设置。 在每个实验组合中，由同一个模型写入存储（Mem0、Zep）并回答所有问题，或由该模型驱动智能体。
Photos. Some LoCoMo turns share a photo, and the official file holds a caption of it next to the turn text. The agents get the official file, captions included; the memory methods were given the turn text only. 116 of the 419 turns share a photo, and 64 of the 152 questions have evidence in such a turn. Section 5 measures what this changes.
照片。 LoCoMo 的一些对话轮次分享了照片，官方文件在对应轮次文本的旁边保存了照片描述。 智能体获得的是包含照片描述的官方文件；记忆方法仅获得轮次文本。 419 轮交流中有 116 轮分享了照片，152 个问题中有 64 个问题的证据来自这样的轮次。 第 5 节 衡量了这会造成什么影响。
Measurement. A rollout is one run over the conversation: the build, then the 152 questions. GPT-4o-mini grades every answer with Mem0's 2025 judge rubric; empty answers count as wrong. Cost is the list price of every model call, LLM and embedding (text-embedding-3-small), with the judge excluded. A turn is one LLM call: one per question for the memory methods, one per step of an agent's loop, and Mem0's and Zep's calls to build their store. Tokens are those calls' input and output tokens, cached input included; embeddings are not counted. Each cell runs once.
测量。 一次完整评测（rollout）是在该对话上运行一遍：先构建记忆，再回答 152 个问题。 GPT-4o-mini 使用 Mem0 的 2025 年评判准则为每个答案评分；空答案视为错误。 成本按每次模型调用的公开标价计算，包括 LLM 和嵌入模型（text-embedding-3-small），但不包括评判模型。 一个调用轮次（turn）对应一次 LLM 调用：记忆方法每个问题调用一次，智能体循环的每一步调用一次，还包括 Mem0 和 Zep 为构建其存储而进行的调用。 token 数是这些调用的输入与输出 token 总数，包含缓存输入；不计入嵌入模型的 token。 每个实验组合运行一次。
3. Agent memory is a costly patch for a weak model
With GPT-4o-mini, Mem0 scores highest. It scores 78%, against 74% for both full context and BM25; Zep scores 43%. The agents score 63% (Pi), 28% (Codex) and 29% (Claude Code): Codex opens the conversation file for three questions in four and Claude Code almost never, and otherwise they answer from the question alone.
使用 GPT-4o-mini 时，Mem0 得分最高。 其得分为 78%，而完整上下文和 BM25 均为 74%；Zep 为 43%。 智能体的得分分别为 63%（Pi）、28%（Codex）和 29%（Claude Code）：Codex 在四分之三的问题上会打开对话文件，而 Claude Code 几乎从不打开；其余情况下，它们仅根据问题本身作答。
With the two stronger models, the patch stops helping. With GPT-6 Luna, full context scores 92%, BM25 82%, Mem0 80% and Zep 51%; Zep's graph holds 128 facts from the 19 sessions, against 195 with Claude Opus 5.5. With Claude Opus 5.5, Mem0 and full context both score 95% and Zep 87%, and the agents score highest: Pi 99%, Codex 98% and Claude Code 97%, a lead that comes mostly from questions about photos (Section 5). Earlier work also finds that plain retrieval keeps up with elaborate memory and that rankings between memory systems change with the model.
使用两种更强的模型时，这种补丁便不再有帮助。 使用 GPT-6 Luna 时，完整上下文得分为 92%，BM25 为 82%，Mem0 为 80%，Zep 为 51%；Zep 的图谱从 19 次会话中保存了 128 条事实，而使用 Claude Opus 5.5 时则为 195 条。 使用 Claude Opus 5.5 时，Mem0 和完整上下文的得分均为 95%，Zep 为 87%，而智能体得分最高：Pi 为 99%，Codex 为 98%，Claude Code 为 97%；这一领先优势主要来自与照片有关的问题（第 5 节）。 此前的研究也发现，简单检索能够与精心设计的记忆系统相媲美，而且记忆系统之间的排名会随模型而变化。
The patch is not free. Mem0's build alone costs more than full context's whole rollout with the same model: $0.56 against $0.43 with GPT-4o-mini, $0.45 against $0.29 with GPT-6 Luna and $25.88 against $17.84 with Claude Opus 5.5 (Figure 2). Its questions are cheaper, so the build would pay for itself after 372, 545 and 596 questions on the same history; LoCoMo asks 152. Mem0 and Zep are never on the frontier of Figure 1, which runs through BM25 and full context with GPT-6 Luna and Pi with Claude Opus 5.5. Paying to build memory that a stronger model does not need is a memory tax.
这种补丁并非没有代价。 使用相同模型时，Mem0 仅构建记忆的成本就超过了完整上下文方法一整次评测的成本：使用 GPT-4o-mini 时为 0.56 美元对 0.43 美元，使用 GPT-6 Luna 时为 0.45 美元对 0.29 美元，使用 Claude Opus 5.5 时为 25.88 美元对 17.84 美元（图2）。 它回答问题的成本较低，因此在同一段历史上分别回答 372、545 和 596 个问题之后，构建成本才能收回；LoCoMo 只提出了 152 个问题。 Mem0 和 Zep 都未出现在 图1 的前沿上；该前沿依次经过搭配 GPT-6 Luna 的 BM25 和完整上下文，以及搭配 Claude Opus 5.5 的 Pi。 付费构建更强模型并不需要的记忆，就是一种记忆税。
4. Agent memory has a ceiling
The model that writes the store limits every reader. Mem0 and Zep write their store with the same model that later reads it. Retrieval calls no language model, so the reader can be swapped: each model answers from the retrievals of the others' stores (Figure 3, Table 3).
写入存储的模型限制了每一个读取模型。 Mem0 和 Zep 使用同一个模型写入存储，并在之后读取它。 检索过程不调用语言模型，因此可以替换读取模型：每个模型都使用从其他模型所写存储中检索到的内容来回答问题（图3、附表3）。
A stronger reader does not recover what the writer left out. On Zep's GPT-4o-mini graph, GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5 score 43%, 43% and 45%; on the graph Claude Opus 5.5 wrote, 66%, 71% and 87%. Mem0 keeps the order with smaller gaps: 78%, 83% and 84% from GPT-4o-mini's store, 89%, 91% and 95% from Claude Opus 5.5's. This agrees with work where verbatim chunks beat extracted facts on long conversations.
更强的读取模型无法恢复写入模型遗漏的内容。 在 Zep 使用 GPT-4o-mini 构建的图谱上，GPT-4o-mini、GPT-6 Luna 和 Claude Opus 5.5 的得分分别为 43%、43% 和 45%；在 Claude Opus 5.5 写入的图谱上，得分分别为 66%、71% 和 87%。 Mem0 保持了相同的排序，但差距较小：读取 GPT-4o-mini 写入的存储时，得分分别为 78%、83% 和 84%；读取 Claude Opus 5.5 写入的存储时，得分分别为 89%、91% 和 95%。 这与长对话中原文片段优于抽取事实的研究结果一致。
5. With Claude Opus 5.5, Pi tops the frontier
Pi with Claude Opus 5.5 is the top of the frontier in Figure 1. No other cell scores higher: Pi scores 99% for $6.95 a rollout, while full context with the same model scores 95% for $17.84. Pi reads about 10,000 input tokens a question, 7,900 of them cached, against 28,900 for full context. Codex and Claude Code score 98% and 97% for $25.41 and $28.41; HarnessTax explains why the same model costs so differently across harnesses.
搭配 Claude Opus 5.5 的 Pi 位于 图1 前沿的顶端。 没有其他实验组合的得分更高：Pi 每次完整评测花费 6.95 美元，得分为 99%；使用相同模型的完整上下文方法花费 17.84 美元，得分为 95%。 Pi 每个问题读取约 10,000 个输入 token，其中 7,900 个来自缓存；完整上下文则为 28,900 个。 Codex 和 Claude Code 的得分分别为 98% 和 97%，成本分别为 25.41 美元和 28.41 美元；HarnessTax 解释了为什么同一个模型在不同智能体运行框架中的成本会有如此大的差异。
Its one miss is LoCoMo's mistake. The question asks what kind of painting Caroline shared with Melanie on 13 October 2023, and the answer key says an abstract painting with blue streaks. In the conversation it is Melanie who shares that painting; Caroline shares a drawing. Pi says so and is marked wrong: “Caroline shared a drawing, not a painting. … The paintings shared that day (a sunset and an abstract piece) were Melanie's.” With Claude Opus 5.5, full context, Codex and Claude Code point out the same mistake and are marked wrong too. On the other 151 questions, Pi with Claude Opus 5.5 is always right.
它唯一的一道错题来自 LoCoMo 的错误。 该问题询问 Caroline 在 2023 年 10 月 13 日与 Melanie 分享了什么类型的绘画作品，标准答案是一幅带有蓝色条纹的抽象画。 在对话中，分享那幅绘画作品的其实是 Melanie；Caroline 分享的是一幅画（drawing）。 Pi 指出了这一点，却被判为错误：“Caroline 分享的是一幅画（drawing），而不是一幅绘画作品（painting）。……那天分享的绘画作品（一幅日落画和一幅抽象画）都是 Melanie 的。” 使用 Claude Opus 5.5 时，完整上下文、Codex 和 Claude Code 也指出了同一个错误，同样被判错。 对于其余 151 个问题，搭配 Claude Opus 5.5 的 Pi 全部答对。
Photo captions account for most of Pi's lead over full context. With Claude Opus 5.5, Pi answers seven questions that full context misses and misses none that it answers. Six of the seven have their evidence in a turn that shares a photo, whose caption Pi could read and full context could not (Section 2); so do all six such questions for Codex and all four for Claude Code. On the 88 questions without photo evidence, Pi scores 100%, and full context, Mem0, Codex and Claude Code 99% each (Table 4). With the same input, then, Pi matches full context for less than half its cost. Whether searching the record answers more than reading all of it, this study does not settle.
照片描述解释了 Pi 相对完整上下文的大部分领先优势。 使用 Claude Opus 5.5 时，Pi 答对了完整上下文答错的七个问题，而且没有答错任何完整上下文答对的问题。 这七个问题中有六个的证据来自分享了照片的轮次，Pi 能够读取其照片描述，而完整上下文不能（第 2 节）；Codex 的全部六个此类问题和 Claude Code 的全部四个此类问题也都如此。 在证据不涉及照片的 88 个问题上，Pi 的得分为 100%，完整上下文、Mem0、Codex 和 Claude Code 的得分均为 99%（附表4）。 因此，在输入相同的情况下，Pi 以不到完整上下文一半的成本取得了与之相当的表现。 搜索记录是否比读取全部记录能够答对更多问题，这项研究尚未给出定论。
6. Discussion and limitations
Our results suggest that agent memory is a costly patch for a weak model and caps what any reader can answer. On this LoCoMo conversation, Mem0 helps GPT-4o-mini but not GPT-6 Luna or Claude Opus 5.5, a stronger reader does not recover what the writer left out, and with Claude Opus 5.5 Pi answers every sound question for less than full context costs.
我们的结果表明，智能体记忆是弱模型的一种昂贵补丁，并且会限制任何读取模型所能回答的内容。 在 LoCoMo 的这段对话上，Mem0 对 GPT-4o-mini 有帮助，对 GPT-6 Luna 或 Claude Opus 5.5 则没有；更强的读取模型无法恢复写入模型遗漏的内容；而使用 Claude Opus 5.5 时，Pi 以低于完整上下文的成本答对了所有标准答案正确的问题。
The findings rest on a small study, and these limits apply:
这些发现基于一项小规模研究，存在以下局限：
- Scale. One conversation of LoCoMo's ten, and one run per cell. Differences of a few questions may not hold in a second run.
- Grading. A GPT-4o-mini judge with a lenient rubric, and at least one wrong answer key.
- Photo captions. The agents could read photo captions that the memory methods were not given (Section 5). This touches comparisons between agents and memory methods, not those among the memory methods.
- Context length. The history fits in every model's context window, so full context is always an option. Results may differ on other benchmarks such as LongMemEval and on histories that do not fit.
- Settings and prices. Mem0 and Zep run with their default extraction and retrieval, and other settings may do better. Costs are list prices and depend on each provider's caching.
- 规模。 仅使用 LoCoMo 十段对话中的一段，每个实验组合只运行一次。几个问题的差异可能在第二次运行中就不再成立。
- 评分。 使用 GPT-4o-mini 作为评判模型，评分准则较为宽松，而且至少存在一个错误的标准答案。
- 照片描述。 智能体能够读取记忆方法未获得的照片描述（第 5 节）。这会影响智能体与记忆方法之间的比较，但不影响记忆方法彼此之间的比较。
- 上下文长度。 这段历史记录能够放入每个模型的上下文窗口，因此完整上下文始终是一个可选方案。在 LongMemEval 等其他基准上，或当历史记录超出上下文窗口时，结果可能有所不同。
- 设置与价格。 Mem0 和 Zep 使用默认的抽取和检索设置，其他设置可能表现更好。成本按公开标价计算，并取决于各提供商的缓存机制。
Memory systems and harnesses can also be searched for automatically, but what is found is still fixed before the question arrives. More broadly, a strong model may need less designed memory and better tools: exact, semantic and vector search over the raw record, driven by the model itself. Progress may then look less like another memory system and more like ripgrep after grep, or zvec-grep after ripgrep. Designed memory may survive as a cost trade-off when many questions share one history (Figure 2).
记忆系统和智能体运行框架也可以通过自动搜索获得，但搜索得到的方案仍然是在问题到来之前就固定下来的。 更广泛地说，强模型可能更少需要专门设计的记忆，而更需要优质的工具：由模型自身驱动，对原始记录进行精确搜索、语义搜索和向量搜索。 因此，进步的形式可能更像是 grep 之后的 ripgrep，或 ripgrep 之后的 zvec-grep，而不是又一个记忆系统。 当许多问题共享同一段历史时，专门设计的记忆可能仍会作为一种成本权衡方案而存在（图2）。