---

ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory

MemoryAgent300+ICLR 2026CCF-A伊利诺伊大学厄巴纳-香槟分校Google Cloud AI Research耶鲁大学

Ouyang S, Yan J, Hsu I H, et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory[C]//The Fourteenth International Conference on Learning Representations. 2026.

https://openreview.net/forum?id=jL7fwchScm

https://github.com/google-research/reasoning-bank

---

ReasoningBank：通过推理记忆扩展智能体的自演化能力

Abstract

With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time.

随着大语言模型智能体越来越多地被用于持久的真实世界角色，它们会自然地遇到连续不断的任务流。 然而，一个关键局限在于，它们无法从累积的交互历史中学习，因而不得不丢弃有价值的洞见并重复过去的错误。 我们提出 ReasoningBank，这是一种新颖的记忆框架，能够从智能体自行判断的成功与失败经验中提炼可泛化的推理策略。 在测试时，智能体从 ReasoningBank 中检索相关记忆以指导交互，随后再把新的学习成果整合回去，从而使自身能力随时间不断增强。

Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience. By allocating more compute to each task, the agent generates abundant, diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. The better memory in turn guides more effective scaling, establishing a powerful synergy between memory and test-time scaling.

在这一强大经验学习器的基础上，我们进一步提出记忆感知测试时扩展（MaTTS），通过扩大智能体的交互经验来加速学习过程并提高其多样性。 通过为每个任务分配更多计算资源，智能体会生成丰富且多样的经验，为合成更高质量的记忆提供充分的对比信号。 更好的记忆反过来会引导更有效的扩展，由此在记忆与测试时扩展之间建立强大的协同作用。

Across web browsing and software engineering benchmarks, ReasoningBank consistently outperforms existing memory mechanisms that store raw trajectories or only successful task routines, improving both effectiveness and efficiency; MaTTS further amplifies these gains. These findings establish memory-driven experience scaling as a new scaling dimension, enabling agents to self-evolve with emergent behaviors naturally arise.

在网页浏览与软件工程基准上，ReasoningBank 持续优于存储原始轨迹或仅存储成功任务流程的现有记忆机制，同时提高效果与效率；MaTTS 则进一步放大了这些收益。 这些发现将记忆驱动的经验扩展确立为一种新的扩展维度，使智能体能够在自然涌现行为的同时实现自演化。

1. Introduction

The rapid advancement of large language models (LLMs) has significantly accelerated the development of interactive LLM agents, which are crucial in tackling complex real-world tasks that require multi-turn interactions with environments. These agents have demonstrated great potential across diverse scenarios, including web browsing, computer use, and scientific discovery.

大语言模型（LLM）的快速发展显著加速了交互式 LLM 智能体的发展，而这类智能体对于解决需要与环境进行多轮交互的复杂真实世界任务至关重要。 这些智能体已经在多种场景中展现出巨大潜力，包括网页浏览、计算机使用和科学发现。

As these agents are increasingly deployed in persistent, long-running roles, they naturally encounter a continuous stream of tasks and interactions. However, a critical limitation persists: they largely fail to learn from this accumulated experience. By approaching each new task in isolation, they are doomed to (i) repeat similar errors observed in the past, (ii) discard valuable insights gained from related problems, and, most importantly, (iii) lack self-evolving capabilities that make the agent system more capable over time. This phenomenon highlights the necessity of building memory-aware agent systems that could learn from their past experiences.

随着这些智能体越来越多地被部署到持久、长期运行的角色中，它们会自然地遇到连续不断的任务与交互。 然而，一个关键局限仍然存在：它们基本无法从这些累积经验中学习。 由于孤立地处理每个新任务，它们注定会：(i) 重复过去出现过的相似错误；(ii) 丢弃从相关问题中获得的宝贵洞见；最重要的是，(iii) 缺乏让智能体系统随时间不断增强能力的自演化能力。 这一现象凸显了构建记忆感知智能体系统的必要性，使其能够从过去的经验中学习。

Recent efforts on agent memory have primarily focused on storing past interactions for reuse. However, these approaches are often limited to leveraging raw trajectories or successful routines (i.e., workflows, procedures). They suffer from two fundamental drawbacks. First, they lack the ability to distill higher-level, transferable reasoning patterns. Second, by over-emphasizing successful experiences, they leave the valuable lessons from an agent's own failures largely underexplored. Consequently, existing memory designs often remain limited to passive record-keeping rather than providing actionable, generalizable guidance for future decisions.

近期关于智能体记忆的研究主要聚焦于存储过去的交互以供复用。 然而，这些方法通常局限于利用原始轨迹或成功流程（即工作流、过程）。 它们存在两个根本缺陷。 第一，它们缺乏提炼更高层次、可迁移推理模式的能力。 第二，由于过度强调成功经验，它们基本没有充分探索智能体自身失败所蕴含的宝贵教训。 因此，现有记忆设计通常仍局限于被动记录，而不能为未来决策提供可行动、可泛化的指导。

图1：ReasoningBank induces reusable reasoning strategies, making memory items more transferable for future use. This enables agents to continuously evolve and achieve higher cumulative success rates than the ‘No Memory’ baseline on the WebArena-Admin subset.

To bridge this gap, we propose ReasoningBank, a novel memory framework for agent systems. ReasoningBank distills and organizes memory items from both successful and failed experiences judged by the agent itself without ground-truth labels. As shown in Figure 1, it captures not only effective strategies from successes but also crucial preventative lessons from failures, abstracting them into a collection of actionable principles. This process operates in a closed loop: when facing a new task, the agent retrieves relevant memories from ReasoningBank to guide its actions. Afterward, the new experience is analyzed, distilled, and consolidated back into the ReasoningBank, allowing the agent to continuously evolve and improve its strategic capabilities.

为弥合这一差距，我们提出 ReasoningBank，一种面向智能体系统的新颖记忆框架。 ReasoningBank 从由智能体自身在没有真实标签的情况下判断的成功与失败经验中提炼并组织记忆项。 如图1所示，它不仅从成功中捕捉有效策略，也从失败中提取关键的预防性教训，并将其抽象为一组可行动原则。 这一过程以闭环方式运行：面对新任务时，智能体从 ReasoningBank 检索相关记忆以指导行动。 随后，新经验会被分析、提炼并巩固回 ReasoningBank，使智能体能够持续演化并提高其策略能力。

With ReasoningBank as a strong experience learner, we study experience scaling to establish a powerful synergy between memory and test-time scaling. Instead of scaling experience through breadth by adding more tasks, we focus on scaling experience through depth by tackling each single task with more exploration. We introduce memory-aware test-time scaling (MaTTS) in both parallel and sequential settings, which generates diverse exploration to provide contrastive signals, enabling ReasoningBank to synthesize better memories. It creates a synergy between memory and test-time scaling: high-quality memory steers the scaled exploration toward more promising paths, while the rich experiences generated forge even stronger memories. This positive feedback loop positions memory-driven experience scaling as a new scaling dimension for agents.

以 ReasoningBank 作为强大的经验学习器，我们研究经验扩展，以建立记忆与测试时扩展之间的强大协同作用。 我们不通过增加更多任务从广度上扩展经验，而是通过对每个任务进行更多探索，从深度上扩展经验。 我们在并行和顺序两种设置中提出记忆感知测试时扩展（MaTTS），通过生成多样化探索来提供对比信号，使 ReasoningBank 能够合成更好的记忆。 它在记忆与测试时扩展之间形成协同作用：高质量记忆将扩展探索引向更有希望的路径，而生成的丰富经验又会锻造出更强的记忆。 这一正反馈循环将记忆驱动的经验扩展定位为智能体的一种新扩展维度。

We conduct extensive experiments on challenging benchmarks for web browsing (WebArena, Mind2Web) and software engineering (SWE-Bench-Verified). We demonstrate that ReasoningBank outperforms baselines in both effectiveness (up to 20% relative improvement, Table 1) and efficiency (up to 16% fewer interaction steps, Table 1). Additionally, ReasoningBank synergizes best with MaTTS, making it an essential component for memory-driven experience scaling.

我们在网页浏览（WebArena、Mind2Web）和软件工程（SWE-Bench-Verified）的高难度基准上进行了广泛实验。 我们证明，ReasoningBank 在效果（相对提升最高达 20%，见表1）和效率（交互步骤最多减少 16%，见表1）上都优于基线。 此外，ReasoningBank 与 MaTTS 的协同效果最佳，使其成为记忆驱动经验扩展的关键组件。

Our contributions are threefold: (1) We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from both successful and failed experiences, beyond prior work that primarily stores raw trajectories or success-only routines. (2) We introduce MaTTS that establishes a powerful, bidirectional synergy between memory and test-time scaling, with memory-driven experience as a new scaling dimension. (3) We conduct extensive experiments on web browsing (WebArena, Mind2Web) and software engineering (SWE-Bench-Verified) tasks. We demonstrate that our approaches not only outperform baselines in effectiveness (up to 20% relative improvement) and efficiency (up to 16% fewer interactions), but also uniquely learn from failures and enable agents to develop increasingly complex, emergent reasoning strategies over time.

我们的贡献有三点：(1) 我们提出 ReasoningBank，这是一种新颖的记忆框架，能够从成功和失败经验中提炼可泛化的推理策略，超越主要存储原始轨迹或仅存储成功流程的以往工作。 (2) 我们提出 MaTTS，在记忆与测试时扩展之间建立强大的双向协同作用，并将记忆驱动的经验作为新的扩展维度。 (3) 我们在网页浏览（WebArena、Mind2Web）和软件工程（SWE-Bench-Verified）任务上进行了广泛实验。 我们证明，我们的方法不仅在效果（相对提升最高达 20%）和效率（交互次数最多减少 16%）上优于基线，还能独特地从失败中学习，使智能体能够随时间发展出越来越复杂的涌现推理策略。

2. Related Work

Memory for LLM Agents. Memory has emerged as an essential module in modern agent systems to enhance their performance by utilizing past information. Existing memory systems organize and store information in various forms, including plain text, latent embeddings, and structured graphs. Beyond memory content, those methods usually involve retrieval mechanisms (e.g., semantic search) with memory management strategies (e.g., updating). More recently, with the growing development of reinforcement learning (RL) in LLM agents, RL has also been leveraged for memory management in agent systems.

LLM 智能体的记忆。 记忆已经成为现代智能体系统中的关键模块，通过利用过去的信息来增强系统性能。 现有记忆系统以多种形式组织和存储信息，包括纯文本、潜在嵌入和结构化图。 除记忆内容外，这些方法通常还结合检索机制（例如语义搜索）与记忆管理策略（例如更新）。 最近，随着强化学习（RL）在 LLM 智能体中的发展，RL 也被用于智能体系统的记忆管理。

While most efforts primarily emphasize personalization and long-context management, this paper falls in the research line of learning from past experiences as memory, which is a critical aspect for developing self-evolving agent systems. Different from previous works that emphasize reusing successful trajectories, procedural workflows, or instance-level concepts, ReasoningBank stores high-level strategies and reasoning hints. By abstracting experiences into reusable reasoning units, ReasoningBank enables agents to generalize not only from successful cases but also by learning from failures, providing richer guidance for test-time learning.

虽然大多数工作主要强调个性化和长上下文管理，但本文属于将过去经验作为记忆进行学习的研究路线，这是开发自演化智能体系统的关键方面。 不同于强调复用成功轨迹、过程式工作流或实例级概念的以往工作，ReasoningBank 存储高层策略与推理提示。 通过把经验抽象为可复用的推理单元，ReasoningBank 使智能体不仅能从成功案例中泛化，也能通过失败进行学习，从而为测试时学习提供更丰富的指导。

Agent Test-Time Scaling. Test-time scaling (TTS) has demonstrated strong effectiveness and has become a widely adopted practice in end-to-end problem-solving, such as coding and math reasoning, where methods including best-of-N, beam search, and leveraging verifiers are commonly employed. However, its application to multi-turn interactive scenarios, particularly agentic tasks, remains underexplored. Existing works mainly adapt the lessons learned from reasoning tasks and scale different dimensions of agentic systems, including the search space for each action, the number of agents in multi-agent systems, and the number of interactions with the environment.

智能体测试时扩展。 测试时扩展（TTS）已经展现出很强的有效性，并成为端到端问题求解中广泛采用的实践，例如编程和数学推理；其中通常使用 Best-of-N、束搜索和验证器等方法。 然而，它在多轮交互场景，尤其是智能体任务中的应用仍未得到充分探索。 现有工作主要借鉴推理任务中的经验，并扩展智能体系统的不同维度，包括每个动作的搜索空间、多智能体系统中的智能体数量，以及与环境的交互次数。

We found that none of these efforts considers the role of agent memory in scaling, where an agent can learn from past experiences to guide future decisions. Our work extends this line of research by introducing memory-aware test-time scaling (MaTTS). As we will show in our empirical results (Sections 4.3 and 4.4), memory offers benefits beyond mere computational scaling, where memory and scaling synergistically work towards better performance.

我们发现，这些工作都没有考虑智能体记忆在扩展中的作用，即智能体可以从过去的经验中学习以指导未来决策。 我们的工作通过提出记忆感知测试时扩展（MaTTS）推进了这一研究方向。 正如我们将在实证结果（第 4.3 节和第 4.4 节）中展示的，记忆带来的收益超越单纯的计算扩展，记忆与扩展会协同作用以获得更好的性能。

3. Methodology

In this section, we introduce the problem setup (Section 3.1), and present our proposed ReasoningBank (Section 3.2), based on which we further develop memory-aware test-time scaling (MaTTS) (Section 3.3).

在本节中，我们介绍问题设置（第 3.1 节）和所提出的 ReasoningBank（第 3.2 节），并在此基础上进一步开发记忆感知测试时扩展（MaTTS，第 3.3 节）。

3.1 Problem Formulation

Agent Configuration. The scope of this work focuses on LLM-based agents. The agent policy ${\pi }_{\mathcal{L}}(\cdot \mid \mathcal{M},\mathcal{A})$ is parameterized by the backbone LLM $\mathcal{L}$, conditioned on a memory module $\mathcal{M}$, and the action space $\mathcal{A}$, denoted as ${\pi }_{\mathcal{L}}$ for short. The agent needs to perform a task via interacting with the environment, which can be viewed as a sequential decision-making process. Formally, the transition function of the environment is defined as $\mathcal{T}(s_{t+1}\mid s_t,a_t)$ where $s_t$ is the state and $a_t$ is the action selected by ${\pi }_{\mathcal{L}}$ at time $t$.

智能体配置。 本工作的范围聚焦于基于 LLM 的智能体。 智能体策略 ${\pi }_{\mathcal{L}}(\cdot \mid \mathcal{M},\mathcal{A})$ 由骨干 LLM $\mathcal{L}$ 参数化，以记忆模块 $\mathcal{M}$ 和动作空间 $\mathcal{A}$ 为条件，简称为 ${\pi }_{\mathcal{L}}$。 智能体需要通过与环境交互来执行任务，这可以视为一个顺序决策过程。 形式上，环境的转移函数定义为 $\mathcal{T}(s_{t+1}\mid s_t,a_t)$，其中 $s_t$ 是状态，$a_t$ 是 ${\pi }_{\mathcal{L}}$ 在时间 $t$ 选择的动作。

We focus on web browsing and software engineering (SWE) tasks. $\mathcal{A}$ is a set of web navigation operations for web browsing and bash commands for SWE tasks, $\mathcal{M}$ is ReasoningBank and initialized as empty. For each given task, the agent generates a trajectory of $(o_{0:t},a_{0:t})$ for $t$ steps, where observation $o_t$ is from the current state $s_t$. Observations are text-based accessibility tree of web pages for web browsing tasks and code snippets for SWE.

我们聚焦于网页浏览和软件工程（SWE）任务。 对于网页浏览，$\mathcal{A}$ 是一组网页导航操作；对于 SWE 任务，它是一组 bash 命令；$\mathcal{M}$ 是 ReasoningBank，并被初始化为空。 对于每个给定任务，智能体生成一条包含 $t$ 个步骤的轨迹 $(o_{0:t},a_{0:t})$，其中观测 $o_t$ 来自当前状态 $s_t$。 在网页浏览任务中，观测是基于文本的网页可访问性树；在 SWE 任务中，观测则是代码片段。

The agent needs to generate an action $a_{t+1}\in \mathcal{A}$ via:

智能体需要通过下式生成动作 $a_{t+1}\in \mathcal{A}$：

$${\pi }_{\mathcal{L}}(o_{0:t},a_{0:t};\mathcal{M},\mathcal{A})⟶a_{t+1}$$

For implementation, the memory module $\mathcal{M}$ contributes relevant memories as additional system instruction for ${\pi }_{\mathcal{L}}$.

在实现中，记忆模块 $\mathcal{M}$ 将相关记忆作为 ${\pi }_{\mathcal{L}}$ 的额外系统指令。

Test-Time Learning. We focus on the test-time learning paradigm where a sequence of task queries $\mathcal{Q}=\{q_1,q_2,\dots ,q_N\}$ arrives in a streaming fashion, i.e., each query is revealed and must be completed sequentially without access to future ones. In this setting, no ground truth is available during test-time, so the agent must continually evolve by only leveraging its own past trajectories and any self-verification without relying on external labels. This streaming setting highlights two key challenges: (i) how to extract and preserve useful memory from past trajectories, and (ii) how to effectively leverage such memory for future queries to avoid redundantly re-discovering already successful strategies or repeating past mistakes.

测试时学习。 我们聚焦于测试时学习范式，其中一系列任务查询 $\mathcal{Q}=\{q_1,q_2,\dots ,q_N\}$ 以流式方式到达，即每个查询都会依次揭示并完成，且无法访问未来查询。 在这一设置中，测试时没有真实标签可用，因此智能体必须仅利用自身过去的轨迹和自我验证，在不依赖外部标签的情况下持续演化。 这一流式设置凸显了两个关键挑战：(i) 如何从过去轨迹中抽取并保存有用记忆；(ii) 如何有效利用这些记忆处理未来查询，避免重复发现已经成功的策略或重复过去的错误。

图2：Overview of ReasoningBank. Experiences are distilled into structured memory items with a title, description, and content. For each new task, the agent retrieves relevant items to interact with the environment and constructs new ones from both successful and failed trajectories, using LLM-as-a-Judge as a signal provider. These items are then consolidated into ReasoningBank, forming a closed-loop memory process.

3.2 ReasoningBank

Past raw trajectories (or experiences), while being comprehensive and original, are often too lengthy and noisy to be directly applied to the current user query. As illustrated in Figure 2, ReasoningBank distills useful strategies and reasoning hints from past experiences into structured memory items, which are then stored in the agent's memory for future reuse.

过去的原始轨迹（或经验）虽然全面且原始，但通常过于冗长、嘈杂，难以直接应用于当前用户查询。 如图2所示，ReasoningBank 将过去经验中的有用策略和推理提示提炼为结构化记忆项，随后存入智能体记忆以供未来复用。

Memory Schema. Memory items in ReasoningBank are designed and induced from past experiences as structured knowledge units that abstract away low-level execution details while preserving transferrable reasoning patterns and strategies. Each memory item specifies three components: (i) a title, which serves as a concise identifier summarizing the core strategy or reasoning pattern; (ii) a description, which provides a brief one-sentence summary of the memory item; and (iii) the content, which records the distilled reasoning steps, decision rationales, or operational insights extracted from past experiences. Together, memory items extracted are both human-interpretable and machine-usable, facilitating efficient usage and integration with agents.

记忆模式。 ReasoningBank 中的记忆项由过去经验设计和归纳而来，作为结构化知识单元抽象掉低层执行细节，同时保留可迁移的推理模式和策略。 每个记忆项包含三个组成部分：(i) 标题，作为概括核心策略或推理模式的简洁标识；(ii) 描述，用一句话简要概括记忆项；(iii) 内容，记录从过去经验中提炼出的推理步骤、决策依据或操作洞见。 这些抽取出的记忆项既可供人类解释，也可由机器使用，从而便于高效使用并与智能体整合。

Integrating ReasoningBank with Agents. An agent ${\pi }_{\mathcal{L}}$ equipped with ReasoningBank can draw upon a curated pool of transferable strategies to guide decision-making. This enables the agent to recall effective insights, avoid previously observed pitfalls, and adapt more robustly to unseen queries. The integration proceeds in three steps: (i) memory retrieval, (ii) memory extraction, and (iii) memory consolidation, as shown in Figure 2.

将 ReasoningBank 与智能体整合。 配备 ReasoningBank 的智能体 ${\pi }_{\mathcal{L}}$ 可以利用经过整理的可迁移策略池来指导决策。 这使智能体能够回忆有效洞见、避开先前观察到的陷阱，并更稳健地适应未见查询。 如图2所示，整合过程包含三个步骤：(i) 记忆检索；(ii) 记忆抽取；(iii) 记忆巩固。

During memory retrieval, the agent queries ReasoningBank with the current query context to identify the top-$k$ relevant experiences and their corresponding memory items using embedding-based similarity search. Retrieved items are injected into the agent's system instruction, ensuring that action prediction from $\mathcal{A}$ is grounded with useful past experiences. When the current query task is completed, we will perform memory extraction to extract new memory items. The first step is to obtain proxy correctness signals for trajectories: we adopt an LLM-as-a-judge to label outcomes as success or failure given the query and trajectory without any ground-truth reference.

在记忆检索期间，智能体使用当前查询上下文查询 ReasoningBank，通过基于嵌入的相似度搜索识别 top-$k$ 相关经验及其对应的记忆项。 检索到的记忆项被注入智能体的系统指令，确保从 $\mathcal{A}$ 预测动作时能够以有用的过去经验为依据。 当前查询任务完成后，我们会执行记忆抽取，以提取新的记忆项。 第一步是为轨迹获取代理正确性信号：给定查询和轨迹，我们采用 LLM-as-a-Judge 在没有任何真实参考的情况下将结果标记为成功或失败。

Based on these signals, we apply different extraction strategies: successful experiences contribute validated strategies, while failed ones supply counterfactual signals and pitfalls that help sharpen guardrails. In practice, we extract multiple memory items for each trajectory/experience as detailed in the appendix. Finally, memory consolidation incorporates these items into ReasoningBank with a simple addition operation, maintaining an evolving repository of memory items. Details are in the appendix.

根据这些信号，我们采用不同的抽取策略：成功经验提供经过验证的策略，而失败经验则提供反事实信号和陷阱，帮助强化防护规则。 在实践中，我们会为每条轨迹或每段经验抽取多个记忆项，细节见附录。 最后，记忆巩固通过简单的添加操作把这些记忆项纳入 ReasoningBank，维护一个持续演化的记忆项存储库。 相关细节见附录。

Together, these steps form a closed-loop process: the agent leverages past experiences, constructs new memory from current tasks, and continually updates its memory, enabling sustained evolvement in test-time learning scenarios.

这些步骤共同形成一个闭环过程：智能体利用过去经验，从当前任务构建新记忆，并持续更新记忆，从而在测试时学习场景中实现持续演化。

3.3 MaTTS: Memory-aware Test-Time Scaling

ReasoningBank enables learning from experiences to translate more experiences into greater improvements. As test-time scaling recently emerged as a powerful strategy for boosting the performance of LLM agents, it shows strong potential by allocating additional inference-time computation to generate abundant exploration histories. A direct combination of ReasoningBank and test-time scaling is depicted in Figure 3(a), where more trajectories are independently converted to more memory items. However, this vanilla form is suboptimal because it does not leverage inherent contrastive signal that arises from redundant exploration on the same problem, which limits the resulting performance advantage brought by test-time scaling.

ReasoningBank 能够从经验中学习，使更多经验转化为更大的改进。 测试时扩展最近已经成为提升 LLM 智能体性能的有力策略，它通过分配额外的推理时计算来生成大量探索历史，展现出很大潜力。 ReasoningBank 与测试时扩展的直接组合如图3(a)所示，其中更多轨迹被独立转换为更多记忆项。 然而，这种普通形式并非最优，因为它没有利用对同一问题进行冗余探索时产生的内在对比信号，从而限制了测试时扩展带来的性能优势。

To address this, we propose Memory-aware Test-Time Scaling (MaTTS), a novel integration of test-time scaling with ReasoningBank. Unlike the vanilla approach, MaTTS deliberately learns from the abundant successful and failure trajectories generated during scaling for more effective memory curation. We design two complementary instantiations for MaTTS, parallel scaling and sequential scaling, as illustrated in Figure 3(b) and Figure 3(c) with detailed implementation in the appendix.

为解决这一问题，我们提出记忆感知测试时扩展（MaTTS），这是一种将测试时扩展与 ReasoningBank 整合的新方法。 不同于普通方法，MaTTS 会有意识地从扩展过程中生成的大量成功与失败轨迹中学习，以更有效地整理记忆。 我们为 MaTTS 设计了两个互补实例：并行扩展与顺序扩展，如图3(b)和图3(c)所示，详细实现见附录。

图3：Comparison of (a) vanilla TTS and MaTTS with (b) parallel scaling, where self-contrast across multiple trajectories curates reliable memory, and (c) sequential scaling, where self-refinement enriches memory with intermediate reasoning signals.

Parallel Scaling. In the parallel setting, we generate multiple trajectories for the same query under the guidance of retrieved memory items. By comparing and contrasting (self-contrast) across different trajectories, the agent can identify consistent reasoning patterns while filtering out spurious solutions. This process enables more reliable memory curation from multiple trials of a single query that promotes diverse exploration.

并行扩展。 在并行设置中，我们在检索到的记忆项指导下，为同一查询生成多条轨迹。 通过在不同轨迹之间进行比较和对照（自对照），智能体可以识别一致的推理模式，同时过滤虚假解法。 这一过程能够从单个查询的多次尝试中更可靠地整理记忆，并促进多样化探索。

Sequential Scaling. We iteratively refines its reasoning within a single trajectory after the initial completion, following the principle of self-refinement. During this process, the intermediate notes generated in self-refinement are also used as valuable signals for memory, since they capture reasoning attempts, corrections, and insights that may not appear in the final solution.

顺序扩展。 在初次完成后，我们遵循自我精炼原则，在单条轨迹内部迭代地精炼推理。 在这一过程中，自我精炼生成的中间笔记也会作为宝贵的记忆信号，因为它们捕捉了最终解答中可能不会出现的推理尝试、修正和洞见。

We define the scaling factor $k$, denoting the number of trajectories for parallel scaling and refinement steps for sequential scaling. Equipped with ReasoningBank, both parallel and sequential strategies become memory-aware, ensuring that the additional computation allocated at test time translates into more transferable and higher-quality memory for future tasks.

我们定义扩展因子 $k$，它在并行扩展中表示轨迹数量，在顺序扩展中表示精炼步骤数。 配备 ReasoningBank 后，并行与顺序策略都具备记忆感知能力，确保测试时分配的额外计算能够转化为面向未来任务、更可迁移且质量更高的记忆。

4. Experiments

4.1 Setup

Following existing work, we conduct experiments on WebArena which features general web navigation across diverse domains, and Mind2Web that tests generalization of agents on versatile operations and environments. We also conduct experiment on SWE-Bench-Verified for repository-level issue-resolving. For comparison, we consider baselines ranging from memory-free agents (No Memory) to trajectory-based memory (Synapse) and workflow-based memory (AWM).

遵循现有工作，我们在 WebArena 上进行实验，该基准涵盖多个领域的通用网页导航；同时使用 Mind2Web 测试智能体在多样操作与环境中的泛化能力。 我们还在 SWE-Bench-Verified 上进行仓库级问题解决实验。 作为比较，我们考虑从无记忆智能体（No Memory）到基于轨迹的记忆（Synapse）和基于工作流的记忆（AWM）等基线。

Our agents are built on Gemini-2.5 and Claude-3.7 models using BrowserGym for web browsing and bash-only for SWE, following ReAct style with default decoding configurations. We evaluate effectiveness (success rate, SR) and efficiency (average steps, AS), with specific metrics varying for each dataset. Full descriptions for datasets, baselines, implementations, and evaluation are in the appendix.

我们的智能体基于 Gemini-2.5 和 Claude-3.7 模型构建：网页浏览使用 BrowserGym，SWE 仅使用 bash；整体遵循 ReAct 风格并采用默认解码配置。 我们评估效果（成功率，SR）和效率（平均步骤数，AS），具体指标随数据集而异。 数据集、基线、实现和评估的完整说明见附录。

表1：Experiment results of ReasoningBank and MaTTS (parallel scaling, k = 5, pass@1) on WebArena benchmark. Success rate (SR ↑) and the number of steps (Step ↓) are reported on 5 subsets for 3 different backbone LLMs.

Models	Shopping		Admin		Gitlab		Reddit		Multi		Overall
	(187)		(182)		(180)		(106)		(29)		(684)
	SR	Step	SR	Step	SR	Step	SR	Step	SR	Step	SR	Step
Gemini-2.5-flash
No Memory	39.0	8.2	44.5	9.5	33.9	13.3	55.7	6.7	10.3	10.0	40.5	9.7
Synapse	40.6	7.0	45.1	9.1	35.6	13.0	59.4	6.5	10.3	10.5	42.1	9.2
AWM	44.4	7.0	46.7	8.8	37.2	13.2	62.3	6.1	3.4	7.7	44.1	9.0
ReasoningBank	49.7	6.1	51.1	8.2	40.6	12.3	67.0	5.6	13.8	8.8	48.8	8.3
+MaTTS	53.0	6.3	53.8	7.6	42.8	11.9	70.8	5.4	17.2	8.0	51.8	7.9
Gemini-2.5-pro
No Memory	45.5	7.6	51.1	8.7	35.0	11.6	71.7	6.0	6.9	8.8	46.7	8.8
Synapse	46.5	6.6	52.2	8.9	38.3	11.3	68.9	5.9	6.9	9.0	47.7	8.5
AWM	48.1	6.4	49.3	9.8	40.0	11.2	68.9	6.4	3.4	9.3	47.6	8.7
ReasoningBank	51.9	6.0	56.6	7.7	44.4	9.8	80.2	5.1	13.8	8.2	53.9	7.4
+MaTTS	54.0	5.9	58.2	7.4	46.7	9.1	83.0	5.3	20.7	7.2	56.3	7.1
Claude-3.7-sonnet
No Memory	38.5	6.1	49.5	8.4	36.7	10.6	53.8	5.5	0.0	11.6	41.7	8.0
Synapse	39.6	5.8	50.5	8.5	38.0	10.0	53.8	6.1	0.0	11.8	42.6	7.9
AWM	39.6	7.2	47.8	9.3	34.6	10.9	52.8	7.0	0.0	12.4	40.8	8.9
ReasoningBank	44.9	5.6	53.3	7.6	41.1	9.5	57.5	5.2	3.4	10.5	46.3	7.3
+MaTTS	47.1	5.8	55.5	7.4	43.3	9.4	60.4	5.0	10.3	9.1	48.8	7.2

4.2 Results of ReasoningBank

Tables 1, 3, 2 summarize the main evaluation results of ReasoningBank on WebArena, Mind2Web, and SWE-Bench-Verified accordingly. We have the following observations.

表1、表3和表2分别总结了 ReasoningBank 在 WebArena、Mind2Web 和 SWE-Bench-Verified 上的主要评估结果。 我们得到以下观察。

ReasoningBank consistently outperforms baselines across LLM backbones on all datasets. Specifically, ReasoningBank improves the overall success rate on WebArena (Table 1) by $+8.3$, $+7.2$, and $+4.6$ with three different backbone LLMs compared to memory-free agents. A similar pattern holds on Mind2Web (Table 3), where ReasoningBank delivers clear gains across cross-task, cross-website, and cross-domain settings, underscoring both the consistency and scalability of its benefits across datasets and model sizes. Results on SWE-Bench-Verified (Table 2) further confirm its robustness. Crucially, unlike baselines such as Synapse and AWM that rely on a narrow, homogeneous memory source derived exclusively from successful trajectories, ReasoningBank employs a superior extraction strategy that is key to its consistent out-performance.

ReasoningBank 在所有数据集和不同 LLM 骨干上都持续优于基线。 具体而言，与无记忆智能体相比，ReasoningBank 在三个不同骨干 LLM 上分别将 WebArena 的总体成功率（表1）提高 $+8.3$、$+7.2$ 和 $+4.6$。 Mind2Web（表3）上也呈现类似模式：ReasoningBank 在跨任务、跨网站和跨领域设置中都带来明显提升，凸显其收益在不同数据集和模型规模上的一致性与可扩展性。 SWE-Bench-Verified（表2）上的结果进一步证实了其稳健性。 关键在于，不同于 Synapse 和 AWM 等仅依赖成功轨迹所形成的狭窄、同质记忆源的基线，ReasoningBank 采用了更优的抽取策略，这是其持续取得更好性能的关键。

表2：Experiment results of ReasoningBank on SWE-Bench-Verified dataset for issue-resolving in a given repository.

Methods	Resolve Rate	AS
Gemini-2.5-flash
No Memory	34.2	30.3
Synapse	35.4	30.7
ReasoningBank	38.8	27.5
Gemini-2.5-pro
No Memory	54.0	21.1
Synapse	53.4	21.0
ReasoningBank	57.4	19.8

ReasoningBank enhances generalization with better transferrable memory across tasks. We also evaluate in challenging generalization settings. On WebArena (Table 1), the Multi subset requires transferring memory across multiple websites, where ReasoningBank achieves a notable gain of $+4.6$ averaged SR over the strongest baseline. In contrast, strong baselines such as AWM fail to provide gains and even degrade in this setting.

ReasoningBank 通过在任务之间提供更好的可迁移记忆来增强泛化能力。 我们还在高难度泛化设置中进行评估。 在 WebArena（表1）上，Multi 子集要求跨多个网站迁移记忆；ReasoningBank 相比最强基线取得了平均 SR $+4.6$ 的显著提升。 相比之下，AWM 等强基线在这一设置中没有带来收益，甚至出现退化。

On Mind2Web (Table 3), which includes cross-task, cross-website, and cross-domain evaluations that impose progressively higher demands, ReasoningBank consistently improves success rates. The gains are especially pronounced in the cross-domain setting, which requires the highest level of generalization. These results demonstrate that memory curated by ReasoningBank is more robust and transferable, enabling agents to generalize effectively across diverse scenarios.

Mind2Web（表3）包含要求逐渐提高的跨任务、跨网站和跨领域评估，ReasoningBank 在这些设置中持续提高成功率。 在要求最高泛化水平的跨领域设置中，提升尤其明显。 这些结果表明，ReasoningBank 整理的记忆更加稳健且更具可迁移性，使智能体能够在多样场景中有效泛化。

表3：Results on Mind2Web benchmark for cross-task, cross-website, and cross-domain generalization test. EA (↑) is short for element accuracy, AF1 (↑) is short for action F1, and SSR (↑) is short for step success rate. SR (↑) is the task-level success rate measuring if all steps are correct.

Models	Cross-Task				Cross-Website				Cross-Domain
	EA	AF1	SSR	SR	EA	AF1	SSR	SR	EA	AF1	SSR	SR
Gemini-2.5-flash
No Memory	46.0	59.1	40.3	3.3	39.8	45.1	31.7	1.7	35.8	37.9	31.9	1.0
Synapse	47.0	59.5	41.2	3.5	40.3	46.0	32.1	1.9	36.3	38.5	32.4	1.1
AWM	46.3	56.1	41.0	3.5	39.1	42.2	31.7	2.1	33.3	36.5	30.1	0.7
ReasoningBank	52.1	60.4	44.9	4.8	44.3	52.6	33.9	2.3	40.6	41.3	36.6	1.6
Gemini-2.5-pro
No Memory	49.3	60.2	44.4	3.5	41.2	49.8	34.8	3.4	37.9	37.7	35.0	1.4
Synapse	50.1	61.0	44.7	3.6	41.8	51.2	35.0	3.2	38.5	39.8	35.6	1.5
AWM	48.6	61.2	44.4	3.7	41.9	47.9	34.8	2.3	37.3	38.1	34.4	1.2
ReasoningBank	53.6	62.7	45.6	5.1	46.1	54.8	36.9	3.8	42.8	45.2	38.1	1.7

ReasoningBank achieves superior efficiency by leveraging past experiences as memory. In addition to higher success rates, ReasoningBank also reduces the number of interaction steps needed to complete tasks, as shown in the Step metric of Table 1 and Table 2. On WebArena, across almost all subsets and backbones, ReasoningBank lowers the average step count by up to $1.4$ compared with “No Memory”, and $1.6$ compared with other memory baselines. The average step on SWE-Bench-Verified is also smaller by saving $2.8$ and $1.3$ steps respectively. This indicates that ReasoningBank enables agents to solve tasks more efficiently by reusing and refining reasoning knowledge, thus avoiding unnecessary or redundant exploration.

ReasoningBank 通过把过去经验作为记忆来获得更高效率。 除了更高的成功率外，ReasoningBank 还减少了完成任务所需的交互步骤数，如表1和表2中的 Step 指标所示。 在 WebArena 上，ReasoningBank 几乎在所有子集和骨干上都降低了平均步骤数：相较于“No Memory”最多减少 $1.4$ 步，相较于其他记忆基线最多减少 $1.6$ 步。 在 SWE-Bench-Verified 上，平均步骤数也分别减少了 $2.8$ 和 $1.3$ 步。 这表明 ReasoningBank 通过复用和精炼推理知识，使智能体能够更高效地解决任务，从而避免不必要或冗余的探索。

4.3 Results of MaTTS

We first present the overall results of MaTTS on WebArena in Table 1, which demonstrate additional strong performance improvement and efficiency gains with ReasoningBank. To study the scaling effect in-depth with both parallel and sequential variants, we experimented MaTTS with Gemini-2.5-flash on Webarena-Shopping subset. To investigate the overall scaling effect, we benchmark with (i) MaTTS w/o memory, which represents the scaling setting without memory mechanism, (ii) MaTTS w/o aggregation, which is equal to Vanilla TTS in Figure 3(a) and (iii) MaTTS to demonstrate the effect with respect to scaling factor $k$.

我们首先在表1中给出 MaTTS 在 WebArena 上的总体结果，结果表明它与 ReasoningBank 结合后带来了额外且显著的性能提升与效率收益。 为了深入研究并行和顺序两种变体的扩展效应，我们在 WebArena-Shopping 子集上使用 Gemini-2.5-flash 进行 MaTTS 实验。 为了研究总体扩展效应，我们设置以下基线：(i) MaTTS w/o memory，表示没有记忆机制的扩展设置；(ii) MaTTS w/o aggregation，等同于图3(a)中的普通 TTS；以及 (iii) MaTTS，用于展示相对于扩展因子 $k$ 的效果。

Notably, $k=1$ is the setting without scaling. For parallel scaling, we compute Best-of-N (BoN) as the final metric detailed in the appendix. Results are shown in Figure 4.

需要注意的是，$k=1$ 表示不进行扩展的设置。 对于并行扩展，我们采用附录中详述的 Best-of-N（BoN）作为最终指标。 结果如图4所示。

Both parallel scaling and sequential scaling boost performance. Increasing $k$ generally improves success rate, confirming the benefit of allocating more inference-time computation. With MaTTS, parallel scaling grows from $49.7$ ($k=1$) to $55.1$ ($k=5$), while sequential scaling rises from $49.7$ to $54.5$. For the baseline of MaTTS w/o memory, the gains are smaller and less consistent (e.g., parallel scaling fluctuates between $39.0$ and $42.2$ and sequential scaling fluctuates between $37.4$ and $40.6$). In contrast, MaTTS enables stronger and more stable improvements across both scaling strategies, highlighting its role in making scaling more effective.

并行扩展和顺序扩展都能提升性能。 增大 $k$ 通常会提高成功率，这证实了分配更多推理时计算的益处。 使用 MaTTS 时，并行扩展从 $49.7$（$k=1$）提高到 $55.1$（$k=5$），顺序扩展则从 $49.7$ 提高到 $54.5$。 对于 MaTTS w/o memory 基线，收益更小且更不稳定（例如，并行扩展在 $39.0$ 到 $42.2$ 之间波动，顺序扩展在 $37.4$ 到 $40.6$ 之间波动）。 相比之下，MaTTS 在两种扩展策略上都能实现更强、更稳定的提升，凸显了它在提高扩展有效性方面的作用。

MaTTS is consistently better than vanilla TTS. With ReasoningBank, MaTTS consistently surpasses MaTTS w/o aggregation (vanilla TTS), showing that memory-aware coordination and aggregation is important. Specifically, at $k=5$, MaTTS achieves $55.1$ in parallel scaling compared with $52.4$ for vanilla TTS, and $54.5$ versus $51.9$ in sequential scaling. These improvements highlight that memory-aware scaling effectively directs the agent toward more promising solutions by synthesizing insights from multiple trajectories or interaction steps to leverage contrastive signals.

MaTTS 始终优于普通 TTS。 在 ReasoningBank 的支持下，MaTTS 始终优于 MaTTS w/o aggregation（普通 TTS），说明记忆感知的协调与聚合十分重要。 具体而言，在 $k=5$ 时，MaTTS 的并行扩展达到 $55.1$，而普通 TTS 为 $52.4$；顺序扩展则分别为 $54.5$ 和 $51.9$。 这些提升表明，记忆感知扩展可以综合多条轨迹或多个交互步骤中的洞见，利用对比信号，有效引导智能体走向更有希望的解法。

图4：Effect of scaling factor k for MaTTS under with ReasoningBank on WebArena-Shopping subset. We compare (a) parallel and (b) sequential test-time scaling.

Sequential scaling shows short-term advantage, but parallel dominates at larger scales for ReasoningBank. With stronger memory mechanisms such as ReasoningBank, sequential refinement brings higher gains at small $k$, but its benefit quickly saturates—once the model either succeeds or fails decisively, further refinements add little new insight. In contrast, parallel scaling continues to provide diverse rollouts that allow the model to critique and improve upon its own generations, leading it to surpass sequential at larger $k$ (e.g., $55.1$ vs. $54.5$ at $k=5$). In contrast, for vanilla TTS which is not equipped with memory module, sequential scaling yields little or even no benefit as scaling goes on, and parallel scaling consistently dominates.

对于 ReasoningBank，顺序扩展具有短期优势，但并行扩展在更大规模下占优。 使用 ReasoningBank 等更强记忆机制时，顺序精炼在较小 $k$ 下带来更高收益，但其收益很快饱和——一旦模型明确成功或失败，进一步精炼几乎不会增加新的洞见。 相比之下，并行扩展会持续提供多样化的展开，使模型能够批评并改进自身生成，因此在更大 $k$ 下超过顺序扩展（例如，$k=5$ 时为 $55.1$ 对 $54.5$）。 对于没有配备记忆模块的普通 TTS，随着扩展继续进行，顺序扩展几乎没有甚至完全没有收益，而并行扩展始终占优。

4.4 Synergy of Memory and Test-Time Scaling

While the previous section establishes the overall effectiveness of MaTTS, we highlight the synergy between memory and TTS in this section. Figure 5 presents a snapshot of MaTTS on the WebArena-Shopping subset with parallel scaling factor $k=5$, where we report both Pass@1 (randomly selected trajectory) and Best-of-5 (BoN). This setting allows us to examine the bidirectional interaction between memory quality and scaling effectiveness.

上一节确立了 MaTTS 的总体有效性，本节则强调记忆与 TTS 之间的协同作用。 图5展示了 MaTTS 在 WebArena-Shopping 子集上、并行扩展因子 $k=5$ 时的一个快照，其中同时报告 Pass@1（随机选择的轨迹）和 Best-of-5（BoN）。 这一设置使我们能够考察记忆质量与扩展有效性之间的双向交互。

图5：Snapshot of MaTTS on WebArena-Shopping subset with different memory mechanisms with k = 5. We compute BoN for all 5 trajectories and Pass@1 with one randomly selected trajectory.

Better memory enables stronger test-time scaling performance. To see how memory improves the effectiveness of scaling, we focus on the BoN results, which directly measure an agent's ability to surface the best outcome among multiple rollouts. As shown by blue bars in Figure 5, the benefit of scaling depends critically on the underlying memory. Without memory, scaling yields slight improvement, with BoN rising only from $39.0$ to $42.2$. Weaker memory mechanisms such as Synapse and AWM provide moderate gains, reaching $44.4$ and $47.6$, respectively.

更好的记忆能够带来更强的测试时扩展性能。 为了观察记忆如何提高扩展有效性，我们聚焦于 BoN 结果，它直接衡量智能体从多次展开中找出最佳结果的能力。 如图5中的蓝色柱所示，扩展收益关键取决于底层记忆。 没有记忆时，扩展仅带来轻微提升，BoN 只从 $39.0$ 上升到 $42.2$。 Synapse 和 AWM 等较弱的记忆机制带来中等收益，分别达到 $44.4$ 和 $47.6$。

In contrast, MaTTS with ReasoningBank delivers the strongest benefit, with BoN climbing from $49.7$ to $55.1$. These results show that high-quality memory directs scaling toward more promising rollouts, ensuring that the additional trajectories are not wasted but converted into higher success rates.

相比之下，结合 ReasoningBank 的 MaTTS 带来最强收益，BoN 从 $49.7$ 提高到 $55.1$。 这些结果表明，高质量记忆会将扩展引向更有希望的展开，确保额外轨迹不会被浪费，而是转化为更高的成功率。

Scaling yields better memory curation. To fairly evaluate how scaling feeds back into memory, we report Pass@1, which measures the average quality of trajectories after memory curation and allows direct comparison with the no-scaling case. The trend is depicted in pick bars and is striking: scaling actually reduces performance for weaker memories, where Synapse slightly increases from $40.6$ to $41.2$, and AWM from $44.4$ to $45.5$. These phenomenon suggest that without strong guidance, the extra rollouts generated by scaling offer diminishing returns.

扩展会带来更好的记忆整理。 为了公平评估扩展如何反馈到记忆，我们报告 Pass@1；它衡量记忆整理后轨迹的平均质量，并允许与无扩展设置直接比较。 粉色柱所示的趋势十分显著：对于较弱记忆，扩展带来的提升很小，Synapse 从 $40.6$ 略升至 $41.2$，AWM 从 $44.4$ 升至 $45.5$。 这些现象表明，如果缺少强有力的指导，扩展生成的额外展开会产生递减收益。

In contrast, ReasoningBank provides far more benefits: Pass@1 rises from $49.7$ to $53.0$, showing that high-quality memory can harness the diversity of scaling to extract constructive contrastive signals. This asymmetry highlights that scaling alone is insufficient; only when paired with good memory mechanism does it contribute to curation of more effective memory, closing the virtuous cycle.

相比之下，ReasoningBank 带来更大收益：Pass@1 从 $49.7$ 提高到 $53.0$，说明高质量记忆能够利用扩展的多样性来提取建设性的对比信号。 这种不对称性表明，仅靠扩展并不充分；只有与良好的记忆机制结合，它才有助于整理出更有效的记忆，从而闭合良性循环。

5. Analysis

We analyze ReasoningBank through several aspects: incorporating failure trajectories, examining emergent strategies, and evaluating efficiency across both successful and failed cases. Additional analyses are presented in the appendix, including but not limited to number of retrieved experiences, additional results on smaller open-source model, and inference cost study.

我们从多个方面分析 ReasoningBank：纳入失败轨迹、考察涌现策略，以及评估成功与失败案例中的效率。 附录还给出了更多分析，包括但不限于检索经验的数量、更小开源模型上的额外结果和推理成本研究。

图6：A case study illustrating emergent behaviors in ReasoningBank through memory items.

Emergent behaviors with ReasoningBank. We find that the strategies in ReasoningBank are not flat or monolithic, but instead evolve over time, exhibiting emergent behaviors that resemble the learning dynamics of RL. As illustrated in a human case study in Figure 6, memory items describing a specific strategy “User-Specific Information Navigation” in ReasoningBank could gradually evolve during test-time learning process. It starts from execution-oriented or procedural strategies (e.g., find navigation links), where the agent follows straightforward action rules. It then progresses to adaptive self-reflections such as re-verifying identifiers to reduce simple mistakes.

ReasoningBank 中的涌现行为。 我们发现，ReasoningBank 中的策略并非扁平或单一，而是会随时间演化，表现出类似 RL 学习动态的涌现行为。 如图6中的人工案例研究所示，ReasoningBank 中描述特定策略“用户特定信息导航”的记忆项可以在测试时学习过程中逐渐演化。 它从执行导向或过程式策略（例如寻找导航链接）开始，此时智能体遵循直接的动作规则。 随后，它发展为自适应自我反思，例如重新验证标识符以减少简单错误。

With more experiences, the same memory item evolves into adaptive checks, where the agent systematically leverages available search or filters to ensure completeness before results. Finally, it eventually matures into compositional strategies such as cross-referencing task requirements and reassessing options. This evolution highlights how ReasoningBank enables agents to refine strategies from low-level actions to high-level reasoning during test-time learning.

随着经验增加，同一记忆项会演化为自适应检查，智能体会系统地利用可用搜索或过滤器，确保结果完整。 最终，它成熟为组合式策略，例如交叉核对任务要求并重新评估选项。 这一演化凸显了 ReasoningBank 如何使智能体在测试时学习期间，把策略从低层动作精炼到高层推理。

图7：Ablation results of incorporating failure trajectories for memory induction.

ReasoningBank makes good use of failure trajectories. Figure 7 compares different memory designs on WebArena-Shopping with Gemini-2.5-flash under two settings: using only successful trajectories versus leveraging both successes and failures. Baseline methods such as Synapse and AWM build memory solely from successful trajectories, and thus are not equipped to benefit from failures. As a result, when failures are added, their performance is limited or even degraded: Synapse increases only from $40.6$ (success only) to $41.7$ (with failures), while AWM drops from $44.4$ to $42.2$.

ReasoningBank 能够充分利用失败轨迹。 图7在 WebArena-Shopping 上使用 Gemini-2.5-flash，对比了两种设置下的不同记忆设计：仅使用成功轨迹，以及同时利用成功与失败轨迹。 Synapse 和 AWM 等基线方法只从成功轨迹构建记忆，因此无法从失败中获益。 因此，加入失败后，它们的性能提升有限，甚至出现退化：Synapse 仅从 $40.6$（仅成功）提高到 $41.7$（包含失败），而 AWM 则从 $44.4$ 降至 $42.2$。

In contrast, the design of ReasoningBank enables distillation of reasoning patterns from both successes and failures, achieving $46.5$ on success-only traces and further improving to $49.7$ when failures are included. This highlights that, unlike baselines, ReasoningBank can transform failures into constructive signals rather than noise, enabling more robust generalization.

相比之下，ReasoningBank 的设计能够从成功和失败中同时提炼推理模式，在仅使用成功轨迹时达到 $46.5$，加入失败后进一步提高到 $49.7$。 这凸显出，不同于基线，ReasoningBank 可以把失败转化为建设性信号而非噪声，从而实现更稳健的泛化。

ReasoningBank exhibits robustness to LLM-as-a-Judge calibration. A critical step in our method is sourcing proxy correctness signals for agent trajectories via an LLM-as-a-Judge. Here we quantitatively calibrate this judge and analyze its robustness to verification noise. We conduct our analysis on the WebArena-Shopping subset, using Gemini-2.5-flash as the judge. We first establish the baseline accuracy by comparing the judge's predictions against ground-truth labels, which we find to be 72.7%.

ReasoningBank 对 LLM-as-a-Judge 的校准具有稳健性。 我们方法中的一个关键步骤，是通过 LLM-as-a-Judge 为智能体轨迹获取代理正确性信号。 这里，我们对该裁判进行定量校准，并分析它对验证噪声的稳健性。 我们在 WebArena-Shopping 子集上开展分析，并使用 Gemini-2.5-flash 作为裁判。 我们首先将裁判预测与真实标签比较，建立基线准确率，结果为 72.7%。

To systematically study the robustness of ReasoningBank to the judge quality, we simulate different levels of verification accuracy. This is achieved by probabilistically correlating the judge's labels with the ground-truth. For example, a 100% accurate verifier uses the ground-truth labels directly. A 90% accurate verifier is simulated by using the correct (ground-truth) label 90% of the time and the incorrect (flipped) label 10% of the time. We extend this simulation down to 50% accuracy, which represents a random-guess baseline for this binary (success/failure) classification task. The results are presented in Figure 8.

为了系统研究 ReasoningBank 对裁判质量的稳健性，我们模拟不同水平的验证准确率。 具体做法是以概率方式让裁判标签与真实标签相关联。 例如，准确率为 100% 的验证器直接使用真实标签。 准确率为 90% 的验证器则通过 90% 的时间使用正确（真实）标签、10% 的时间使用错误（翻转）标签来模拟。 我们将模拟扩展到 50% 准确率，这代表二分类（成功/失败）任务中的随机猜测基线。 结果如图8所示。

图8：Results for SR with respect to simulated LLM-as-a-Judge accuracy.

We observe that the judge's accuracy does not significantly impact the performance of ReasoningBank, as all variants achieve similar success rates within reasonable accuracy range (70%-90%). Intuitively, the 100% (ground-truth) accuracy setting yields the best performance. These findings confirm that ReasoningBank is robust to noise in the verification step.

我们观察到，裁判准确率不会显著影响 ReasoningBank 的性能，因为所有变体在合理准确率范围（70%-90%）内都取得了相近的成功率。 直观上，100%（真实标签）准确率设置取得最佳性能。 这些发现证实 ReasoningBank 对验证步骤中的噪声具有稳健性。

表4：Average number of steps on successful and failed test instances across four WebArena domains. ReasoningBank consistently reduces the number of steps compared to the vanilla baseline, with notably larger reductions on successful instances.

Models	Shopping		Admin		Gitlab		Reddit
	Successful	Failed	Successful	Failed	Successful	Failed	Successful	Failed
No Memory	6.8	8.7	8.4	10.4	8.6	15.7	6.1	7.6
ReasoningBank	4.7 ↓2.1	7.3 ↓1.4	7.0 ↓1.4	9.5 ↓0.9	7.6 ↓1.0	15.5 ↓0.2	5.0 ↓1.1	6.8 ↓0.8

ReasoningBank delivers targeted efficient gains. While the overall number of steps in Table 1 provides a general view of model efficiency, it does not distinguish whether reductions come from successful or failed trajectories. To gain deeper insight, we further separate the analysis into successful and failed test cases, which allows us to understand the source of step reduction: a desirable system should reduce unnecessary exploration when it is on the right track, rather than merely cutting short failed attempts. The results are shown in Table 4.

ReasoningBank 带来有针对性的效率提升。 表1中的总体步骤数给出了模型效率的整体视图，但无法区分减少的步骤来自成功轨迹还是失败轨迹。 为了获得更深入的理解，我们进一步将分析划分为成功和失败测试案例，从而理解步骤减少的来源：理想系统应在沿正确方向前进时减少不必要的探索，而不是仅仅提前终止失败尝试。 结果如表4所示。

We find that ReasoningBank consistently reduces the number of steps across all domains compared to the baseline. More importantly, the reduction is particularly pronounced on successful cases, reaching up to $2.1$ fewer steps (a $26.9\mathrm{\%}$ relative reduction) than on failed ones. This indicates that ReasoningBank primarily helps the agent reach solutions with fewer interactions by strengthening its ability to follow effective reasoning paths rather than simply truncating failed trajectories, which highlight the role of memory in guiding purposeful decision-making and improving efficiency in practice.

我们发现，与基线相比，ReasoningBank 在所有领域中都持续减少步骤数。 更重要的是，这种减少在成功案例上尤其明显，最多减少 $2.1$ 步（相对减少 $26.9\mathrm{\%}$），显著高于失败案例。 这表明，ReasoningBank 主要通过增强智能体沿有效推理路径前进的能力，以更少交互到达解法，而不是简单截断失败轨迹；这凸显了记忆在指导有目的的决策和提高实际效率方面的作用。

6. Conclusion

We introduce ReasoningBank, a memory framework that distills strategy-level reasoning signals from both successes and failures and integrates them into test-time scaling (MaTTS). Extensive experiments show that ReasoningBank consistently improves performance while reducing redundant exploration. Further results reveal a strong synergy between memory and scaling: ReasoningBank guides scaling toward more promising rollouts, while diverse rollouts enrich memory with valuable contrastive signals.

我们提出 ReasoningBank，这是一种从成功与失败中提炼策略级推理信号，并将其整合进测试时扩展（MaTTS）的记忆框架。 广泛实验表明，ReasoningBank 在持续提升性能的同时减少了冗余探索。 进一步结果揭示了记忆与扩展之间的强协同作用：ReasoningBank 将扩展引向更有希望的展开，而多样化展开则以有价值的对比信号丰富记忆。

We also provide analyses of individual components and emergent behaviors. Our findings suggest a practical pathway toward building adaptive and lifelong-learning agents, with additional future directions and limitations in the appendix.

我们还分析了各个组件和涌现行为。 我们的发现为构建自适应、终身学习智能体提供了一条实用途径，更多未来方向与局限见附录。