---

MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

MemoryAgentRLRuntime Learning130+40+上海交通大学西安电子科技大学新加坡国立大学上海创智学院MemTensor中国科学技术大学

Zhang S, Wang J, Zhou R, et al. MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory[J]. arXiv preprint arXiv:2601.03192v2, 2026.

https://github.com/MemTensor/MemRL

---

MemRL：通过情景记忆上的运行时强化学习实现自演化智能体

Abstract

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates.

人类智能的标志，是通过从过去经验中学习来掌握新技能的自演化能力。 然而，当前 AI 智能体难以模拟这种自演化：微调计算成本高且容易发生灾难性遗忘，而现有基于记忆的方法依赖被动语义匹配，常常检索到噪声。 为解决这些挑战，我们提出 MemRL，这是一种通过情景记忆上的强化学习而演化的非参数方法。 通过将稳定推理与可塑记忆解耦，MemRL 采用 Two-Phase Retrieval 机制，通过环境反馈过滤噪声并识别高效用策略。 在 HLE、BigCodeBench、ALFWorld 和 Lifelong Agent Bench 上的大量实验表明，MemRL 显著优于最先进基线，证实 MemRL 能够有效调和稳定性-可塑性困境，在不更新权重的情况下实现持续的运行时改进。

1. Introduction

Human intelligence balances cognitive stability and episodic plasticity via Constructive Episodic Simulation, enabling adaptation without rewiring neural circuitry. Despite their reasoning capabilities, current AI agents struggle to emulate this decoupled self-evolution. Specifically, fine-tuning internalizes experience by modifying weights but suffers from computational costs and catastrophic forgetting. Conversely, Retrieval-Augmented Generation (RAG) provides a non-parametric alternative but remains passive, retrieving by semantic similarity rather than utility; this prevents agents from effectively leveraging runtime feedback to distinguish high-value strategies from noise.

人类智能通过建构性情景模拟来平衡认知稳定性和情景可塑性，使其能够在不重连神经回路的情况下适应环境。 尽管当前 AI 智能体具备推理能力，但它们仍难以模拟这种解耦的自演化。 具体而言，微调通过修改权重来内化经验，但会遭遇计算成本和灾难性遗忘问题。 相反，检索增强生成（RAG）提供了一种非参数替代方案，但它仍是被动的，按语义相似性而不是效用进行检索；这阻止智能体有效利用运行时反馈来区分高价值策略和噪声。

图1：MemRL 的概念框架。

This limitation underscores a critical research question: How can we enable an agent to continuously improve its performance after deployment, without compromising the stability of its pre-trained backbone? Our objective is to achieve an agent that evolves with continued usage and rapidly adapts to new tasks after deployment, referred to as Runtime Continuous Learning, all while keeping the backbone model frozen to prevent catastrophic forgetting. To address this challenge, inspired by the human cognitive mechanism of constructive simulation, we propose MemRL, an approach that facilitates self-evolving agents by explicitly decoupling the model’s stable cognitive reasoning from dynamic episodic memory. Figure 1 illustrates the conceptual framework of our proposed MemRL. Drawing on tries-and-errors manner in Reinforcement Learning (RL) to estimate expected experience utilities, we formalize the interaction between the frozen LLM and external memory as a Markov Decision Process (MDP). Unlike traditional methods that optimize the backbone model, MemRL optimizes the policy of memory usage without tuning model weights.

这一局限凸显出一个关键研究问题：我们如何让智能体在部署后持续提升表现，同时不损害其预训练 backbone 的稳定性？ 我们的目标是实现一种会随着持续使用而演化、并在部署后快速适应新任务的智能体，即运行时持续学习，同时保持 backbone 模型冻结以防止灾难性遗忘。 为解决这一挑战，受人类建构性模拟认知机制的启发，我们提出 MemRL，这是一种通过显式解耦模型稳定认知推理与动态情景记忆来促进自演化智能体的方法。 图1展示了我们提出的 MemRL 的概念框架。 借鉴强化学习（RL）中通过试错估计预期经验效用的方式，我们将冻结 LLM 与外部记忆之间的交互形式化为马尔可夫决策过程（MDP）。 不同于优化 backbone 模型的传统方法，MemRL 在不调节模型权重的情况下优化记忆使用策略。

MemRL organizes memory into a structured Intent-Experience-Utility triplet. This structure transforms retrieval from a passive semantic match task into an active decision-making process: Two-Phase Retrieval selects experiences based on their learned Q-values, reflecting expected utility, rather than semantic similarity alone; Utility-Driven Update refines these Q-values through environmental feedback, applying Monte Carlo style updates. This closed-loop cycle enables the agent to distinguish high-value memories from similar noise, effectively learning from both success and failure without high computational cost or catastrophic forgetting risks associated with weight updates. As for experiments, we validate MemRL on four diverse benchmarks, including HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench. Our results demonstrate consistent superiority over baselines, achieving relative improvement in exploration-heavy environments. Our in-depth analysis reveals a strong correlation between learned utility and task success, further confirming MemRL’s effectiveness. In summary, our contributions are threefold:

MemRL 将记忆组织为结构化的 Intent-Experience-Utility 三元组。 这种结构将检索从被动语义匹配任务转变为主动决策过程：Two-Phase Retrieval 根据学得的 Q-value 选择经验，反映预期效用，而不只依赖语义相似性；Utility-Driven Update 通过环境反馈并应用 Monte Carlo 风格更新来细化这些 Q-value。 这一闭环循环使智能体能够把高价值记忆与相似噪声区分开来，在不承担高计算成本或权重更新导致灾难性遗忘风险的情况下，从成功和失败中有效学习。 在实验方面，我们在 HLE、BigCodeBench、ALFWorld 和 Lifelong Agent Bench 等四个多样化基准上验证 MemRL。 我们的结果展示出相较基线的一致优势，并在探索密集型环境中取得相对提升。 我们的深入分析揭示了学得效用与任务成功之间的强相关性，进一步证实了 MemRL 的有效性。 总之，我们的贡献有三点：

• We propose a runtime learning framework using Model-Memory decoupling and Intent-Experience-Utility triplet to reconcile the stability-plasticity dilemma, enabling tuning-free agent learning.
• We introduce MemRL, a non-parametric approach enabling agent self-evolution via Two-Phase Retrieval and Utility-Driven Update.
• We conduct extensive evaluations and provide a rigorous analysis for MemRL’s stability, showing how it ensures task integrity and minimizes forgetting.

• 我们提出一种使用模型-记忆解耦和 Intent-Experience-Utility 三元组的运行时学习框架，用于调和稳定性-可塑性困境，使智能体能够在无需调参的情况下学习。
• 我们引入 MemRL，这是一种通过 Two-Phase Retrieval 和 Utility-Driven Update 实现智能体自演化的非参数方法。
• 我们进行了大量评估，并对 MemRL 的稳定性提供严格分析，展示它如何确保任务完整性并最小化遗忘。

2. Related Works

Runtime Learning. Runtime Learning focuses on the post-deployment improvement of agents through interaction streams rather than offline data, marking a shift toward the ``era of experience''. Unlike Continual Learning or Test-Time Adaptation, which typically update parameters to handle forgetting or distribution shifts, our setting constrains the backbone to remain frozen to ensure stability and efficiency. While recent memory-augmented agents emphasize memory organization, the selection problem---identifying which experiences to reuse under feedback---remains a critical challenge. Drawing from value-aware episodic control, we frame runtime learning as identifying valuable episodes. By using interaction feedback to assign utility, our approach guides retrieval and reuse without weight modification, thereby ensuring sustained improvement.

运行时学习。 运行时学习关注智能体通过交互流而不是离线数据在部署后的改进，标志着向“经验时代”的转变。 不同于通常通过更新参数来处理遗忘或分布漂移的持续学习或测试时自适应，我们的设置约束 backbone 保持冻结，以确保稳定性和效率。 虽然近期记忆增强智能体强调记忆组织，但选择问题，即在反馈下识别哪些经验应该被复用，仍然是一项关键挑战。 借鉴价值感知情景控制，我们将运行时学习表述为识别有价值的情景。 通过使用交互反馈来分配效用，我们的方法在不修改权重的情况下引导检索和复用，从而确保持续改进。

Reinforcement Learning. Reinforcement learning has been widely adopted for LLMs enhancement. A representative paradigm is to construct reward signals from human feedback and optimize the model policy accordingly to align with human preference. Other recent approaches leverage rule-based verifiers to improve LLMs' reasoning capabilities. In parallel, agent-oriented research explores how interaction signals can improve tool use and action decision-making, and investigates mechanisms by which language models execute composite actions in environments. Despite the demonstrated effectiveness of reward-driven optimization, these methods generally place learning in the model parameters or additional parametric modules, and thus do not avoid the cost of online updates or the risk of forgetting. In contrast, our method frames memory usage as a learnable decision problem and applies non-parametric reinforcement learning on memory to bypass the risk.

强化学习。 强化学习已被广泛用于增强 LLM。 一种代表性范式是从人类反馈构造奖励信号，并相应地优化模型策略以对齐人类偏好。 其他近期方法利用基于规则的验证器来提升 LLM 的推理能力。 与此同时，面向智能体的研究探索交互信号如何改进工具使用和动作决策，并研究语言模型在环境中执行复合动作的机制。 尽管奖励驱动优化已经表现出有效性，这些方法通常把学习放在模型参数或额外参数化模块中，因此无法避免在线更新成本或遗忘风险。 相比之下，我们的方法把记忆使用表述为可学习的决策问题，并在记忆上应用非参数强化学习来绕开这一风险。

图2：马尔可夫决策过程下记忆增强决策的示例。在时间步 t，智能体从初始记忆集开始。在时间步 t+1，一个 intent 检索相关过去经验但最初导致失败生成；另一个 intent 成功，其相关经验被加入记忆。在时间步 t+2，前一个 intent 检索到新存储的成功经验并产生成功结果。

Agentic Memory. To avoid the costs of fine-tuning, external memory systems have evolved from a static RAG paradigm to dynamic, governable memory structures. Early agentic memory introduced reflection mechanisms and hierarchical management to handle long context experiences. More recent frameworks have systematized the memory lifecycle, focusing on unified storage and structured indexing for complex tasks. Furthermore, adaptive approaches now explore improving retrieval via feedback-driven updates or automated augmentation. However, except for training additional learnable modules, most existing methods still rely predominantly on semantic similarity or heuristic rules, lacking a rigorous metric to evaluate the actual utility of a memory in maximizing returns. Inspired by cognitive theories of memory reconsolidation, MemRL bridges this gap by formulating retrieval as a value-based decision process, learning robust utility estimates (Q-values) from environmental rewards to distinguish high-value experiences.

智能体记忆。 为避免微调成本，外部记忆系统已经从静态 RAG 范式演化为动态、可治理的记忆结构。 早期智能体记忆引入反思机制和层级管理来处理长上下文经验。 更近的框架对记忆生命周期进行了系统化，聚焦复杂任务中的统一存储和结构化索引。 此外，自适应方法现在探索通过反馈驱动更新或自动增强来改进检索。 然而，除了训练额外可学习模块之外，大多数现有方法仍主要依赖语义相似性或启发式规则，缺少用于评估记忆在最大化回报中实际效用的严格度量。 受记忆再巩固认知理论启发，MemRL 通过把检索表述为基于价值的决策过程来弥合这一差距，从环境奖励中学习稳健的效用估计（Q-value），以区分高价值经验。

3. Problem Formulation

In this section, we formally define the problem of memory-augmented generation and establish the theoretical link between agent policy and memory retrieval. We adopt the formulation of Memory-Based Markov Decision Process (M-MDP), and apply our non-parametric reinforcement learning approach to it. Figure 2 provides an illustrative example of this memory-augmented decision process, showing how retrieval outcomes and memory evolution unfold over multiple time steps.

在本节中，我们正式定义记忆增强生成问题，并建立智能体策略与记忆检索之间的理论联系。 我们采用 Memory-Based Markov Decision Process（M-MDP）的形式化，并将我们的非参数强化学习方法应用于其中。 图2提供了这一记忆增强决策过程的示例，展示检索结果和记忆演化如何在多个时间步中展开。

3.1 Memory-Augmented Agent Policy

To enable the agent to self-evolution, we adopt the M-MDP framework, defined by the tuple $(\mathcal{S},\mathcal{A},P,\mathcal{R},\gamma ,\mathcal{M})$. Here, $\mathcal{S}$ and $\mathcal{A}$ represent state and action spaces, $P$ is the transition dynamics, $\mathcal{R}$ is the reward function of state and action, $\gamma \in [0,1)$ denotes the discount factor, and $\mathcal{M}=(\mathcal{S}\times \mathcal{A}\times \mathbb{R}{)}^{\ast }$ constitutes the evolving memory bank of past experiences. At each step $t$, the agent receives state $s_t$ and leverages ${\mathcal{M}}_t$ to generate a response $a_t$ maximizing the expected reward. The joint policy $\pi (a_t|s_t,{\mathcal{M}}_t)$ is formulated as the marginal probability over all possible retrieved items $m$:

为了使智能体能够自演化，我们采用 M-MDP 框架，该框架由元组 $(\mathcal{S},\mathcal{A},P,\mathcal{R},\gamma ,\mathcal{M})$ 定义。 其中，$\mathcal{S}$ 和 $\mathcal{A}$ 分别表示状态空间和动作空间，$P$ 是转移动态，$\mathcal{R}$ 是状态和动作的奖励函数，$\gamma \in [0,1)$ 表示折扣因子，$\mathcal{M}=(\mathcal{S}\times \mathcal{A}\times \mathbb{R}{)}^{\ast }$ 构成由过去经验组成的演化记忆库。 在每个时间步 $t$，智能体接收状态 $s_t$，并利用 ${\mathcal{M}}_t$ 生成最大化预期奖励的响应 $a_t$。 联合策略 $\pi (a_t|s_t,{\mathcal{M}}_t)$ 被形式化为所有可能检索项 $m$ 上的边缘概率：

$$\begin{array}{rl}\pi (a_t\mid s_t,{\mathcal{M}}_t)& =\sum _{m\in {\mathcal{M}}_t}\mu (m\mid s_t,{\mathcal{M}}_t)p_{LLM}(a_t\mid s_t,m).\end{array}$$

where $\mu (m|s_t,{\mathcal{M}}_t)$ is the Retrieval Policy for selecting memory contexts, and $p_{LLM}(a_t|s_t,m)$ is the Inference Policy parameterized by a frozen LLM. This approach transforms retrieval from a passive match into an active decision process, effectively accounting for the functional utility of $m$ in generating successful outcomes $a_t$. In previous RAG or memory-based agentic paradigms, the retrieval policy $\mu$ is usually determined by a fixed vector similarity metric, e.g., cosine similarity of embeddings. While effective for semantic matching, such policies fail to account for the utility of a memory, i.e., whether retrieving $m$ actually leads to a successful outcome $a_t$.

其中 $\mu (m|s_t,{\mathcal{M}}_t)$ 是用于选择记忆上下文的检索策略，$p_{LLM}(a_t|s_t,m)$ 是由冻结 LLM 参数化的推理策略。 这种方法将检索从被动匹配转变为主动决策过程，有效考虑了 $m$ 在生成成功结果 $a_t$ 中的功能性效用。 在以往 RAG 或基于记忆的智能体范式中，检索策略 $\mu$ 通常由固定向量相似性度量决定，例如 embedding 的余弦相似性。 虽然这类策略对语义匹配有效，但它们无法考虑记忆的效用，也就是检索 $m$ 是否真的会带来成功结果 $a_t$。

图3：MemRL 概览。端到端学习循环：给定查询，智能体从记忆中检索上下文，生成输出，并根据奖励更新记忆值 Q。Two-Phase Retrieval：候选项先通过相似性召回，再用学得的 Q-value 重新排序。Utility Update：使用环境奖励更新 Q，以区分功能性效用和语义相似性。

3.2 Non-Parametric Reinforcement Learning

To overcome static similarity limitations, we operationalize the M-MDP framework by formulating memory retrieval as a value-based decision-making process. Unlike parametric methods optimizing ${\pi }_{LLM}$ via weight updates, we optimize the retrieval policy $\mu (m|s,\mathcal{M})$ directly within the memory space by mapping M-MDP components to a structured Intent-Experience-Utility triplet: From Semantic Matching to Decision Making. We instantiate state $s$ as the User Intent, encapsulated by the current query embedding. Consequently, the action space ${\mathcal{A}}_t$ becomes dynamic and discrete, corresponding to selecting a specific $m$ from the memory bank ${\mathcal{M}}_t$. In this formulation, retrieval is not a passive matching task but a strategic decision step taken to augment the generator's action $a$.

为克服静态相似性局限，我们通过将记忆检索表述为基于价值的决策过程来操作化 M-MDP 框架。 不同于通过权重更新优化 ${\pi }_{LLM}$ 的参数化方法，我们通过把 M-MDP 组件映射到结构化的 Intent-Experience-Utility 三元组，直接在记忆空间中优化检索策略 $\mu (m|s,\mathcal{M})$： 从语义匹配到决策。 我们将状态 $s$ 实例化为 User Intent，由当前查询 embedding 表示。 因此，动作空间 ${\mathcal{A}}_t$ 变为动态且离散的，对应于从记忆库 ${\mathcal{M}}_t$ 中选择特定 $m$。 在这一形式化中，检索不再是被动匹配任务，而是用于增强生成器动作 $a$ 的策略性决策步骤。

Defining Utility via Q-Values. While the agent's executable action $a$ is generated by the policy $\pi$ (as formalized in Sec. 3.1), the quality of this generation is strictly conditioned on the retrieved context. Therefore, we adapt the traditional value function $Q(s,a)$ to the retrieval phase, defining $Q(s,m)$ as the expected utility of the subsequent action $a$ augmented by memory $m$. MemRL learns an optimal retrieval policy ${\mu }^{\ast }$ that maximizes this utility:

通过 Q-value 定义效用。 虽然智能体可执行动作 $a$ 由策略 $\pi$ 生成（如第 3.1 节所形式化），但这一生成的质量严格依赖于检索到的上下文。 因此，我们将传统价值函数 $Q(s,a)$ 调整到检索阶段，把 $Q(s,m)$ 定义为由记忆 $m$ 增强后的后续动作 $a$ 的预期效用。 MemRL 学习一个最大化该效用的最优检索策略 ${\mu }^{\ast }$：

$${\mu }^{\ast }(m\mid s,\mathcal{M})=\arg \underset{m\in \mathcal{M}}{max}Q(s,m).$$

In this view, the Q-value acts as a critic for the retrieval mechanism, distinguishing memories that strategically aid the generator from irrelevant noise that merely shares high semantic similarity. Non-Parametric Learning. Since the retrieval action space is decoupled from LLM generation, we perform learning without modifying model weights. Upon receiving environmental feedback $r$, we update the Q-value via a Temporal-Difference (TD) error:

从这个视角看，Q-value 充当检索机制的 critic，将能在策略上帮助生成器的记忆与仅仅具有高语义相似性的不相关噪声区分开。 非参数学习。 由于检索动作空间与 LLM 生成解耦，我们在不修改模型权重的情况下执行学习。 在接收到环境反馈 $r$ 后，我们通过 Temporal-Difference（TD）误差更新 Q-value：

$$\begin{array}{r}Q(s,m)\leftarrow Q(s,m)+\alpha [r+\gamma maxQ(s^{\prime },m^{\prime })-Q(s,m)],\end{array}$$

or Monte Carlo style rule:

或采用 Monte Carlo 风格规则：

$$Q_{\text{new}}\leftarrow Q_{\text{old}}+\alpha (r-Q_{\text{old}}).$$

where $\alpha$ is the learning rate. The Monte Carlo style rule performs as a naturally simplified version of the TD error by setting $s^{\prime }$ as a terminal state to balance complexity and performance, sharing a similar one-step MDP formulation with DeepSeek-R1. These manners allow utility estimates to converge to true expected returns over time. By explicitly updating Q-values within the memory structure, MemRL provides a non-parametric learning manner with a theoretical guarantee, enabling agents to self-evolve through interaction.

其中 $\alpha$ 是学习率。 Monte Carlo 风格规则通过将 $s^{\prime }$ 设为终止状态，在复杂度和性能之间取得平衡，可被视为 TD 误差的自然简化版本，并与 DeepSeek-R1 共享类似的一步 MDP 形式化。 这些方式使效用估计能够随时间收敛到真实预期回报。 通过在记忆结构内显式更新 Q-value，MemRL 提供了一种带有理论保证的非参数学习方式，使智能体能够通过交互自演化。

4. MemRL

Building upon the M-MDP formulation defined in Section 3, we propose MemRL, a framework that enables frozen LLMs to self-evolve via non-parametric reinforcement learning. Instead of modifying the model weights $\theta$, MemRL optimizes the retrieval policy $\mu (m|s,\mathcal{M})$ within an evolving memory space. As illustrated in Figure 3, the framework consists of three core components: (i) a structured Intent-Experience-Utility memory bank, (ii) a Two-Phase Retrieval mechanism that decouples semantic recall from value-aware selection, and (iii) a Runtime Utility Update rule that stabilizes Q-value estimation.

基于第 3 节定义的 M-MDP 形式化，我们提出 MemRL，这是一个使冻结 LLM 能够通过非参数强化学习进行自演化的框架。 MemRL 不修改模型权重 $\theta$，而是在演化记忆空间中优化检索策略 $\mu (m|s,\mathcal{M})$。 如图3所示，该框架包含三个核心组件：(i) 结构化的 Intent-Experience-Utility 记忆库，(ii) 将语义召回与价值感知选择解耦的 Two-Phase Retrieval 机制，以及 (iii) 稳定 Q-value 估计的 Runtime Utility Update 规则。

4.1 The Intent-Experience-Utility Triplet

To support value-based decision-making, we structure the external memory $\mathcal{M}$ not merely as key-value pairs, but as a set of triplets:

为了支持基于价值的决策，我们将外部记忆 $\mathcal{M}$ 不仅构造为键值对，而是构造为一组三元组：

$$\mathcal{M}=\{(z_i,e_i,Q_i){\}}_{i=1}^{|\mathcal{M}|}.$$

where $z_i$ represents the Intent, $e_i$ stores the raw Experience (e.g., a successful solution trace or trajectory), and $Q_i$ denotes the learned Utility. $Q_i$ approximates the expected return of applying experience $e_i$ to intents similar to $z_i$, serving as the critic in RL.

其中 $z_i$ 表示 Intent，$e_i$ 存储原始 Experience（例如成功解法轨迹），$Q_i$ 表示学得的 Utility。 $Q_i$ 近似表示将经验 $e_i$ 应用于与 $z_i$ 相似的 intent 时的预期回报，并在 RL 中充当 critic。

4.2 From Semantic Recall to Value-Aware Selection

Standard RAG systems assume ``similar implies useful,'' but agentic tasks often involve environment-specific routines that generalize poorly. Therefore, MemRL implements a Two-Phase Retrieval strategy. Phase A: Similarity-Based Recall. Given query $s$, we isolate a candidate pool $\mathcal{C}(s)$ of semantically consistent experiences by filtering memory bank $\mathcal{M}$ via cosine similarity and a sparsity threshold $\delta$:

标准 RAG 系统假设“相似意味着有用”，但智能体任务常常涉及泛化能力较差的环境特定流程。 因此，MemRL 实现了一种 Two-Phase Retrieval 策略。 阶段 A：基于相似性的召回。 给定查询 $s$，我们通过余弦相似性和稀疏阈值 $\delta$ 过滤记忆库 $\mathcal{M}$，隔离出语义一致经验的候选池 $\mathcal{C}(s)$：

$$\begin{array}{r}\mathcal{C}(s)={\mathrm{TopK}}_{k_1}(\{i\mid \mathrm{sim}(\mathrm{Emb}(s),\mathrm{Emb}(z_i))>\delta \}).\end{array}$$

where $Emb$ represents the Embedding Model to transfer the raw text to a vector. If $\mathcal{C}(s)=\mathrm{\varnothing }$, MemRL relies solely on the frozen LLM for exploration. Phase B: Value-Aware Selection. To determine the final context ${\mathcal{M}}_{ctx}(s)$, we select top-$k_2$ items from $\mathcal{C}(s)$ using a composite score balancing exploration (similarity) and exploitation (utility $Q$):

其中 $Emb$ 表示用于将原始文本转换为向量的 embedding 模型。 如果 $\mathcal{C}(s)=\mathrm{\varnothing }$，MemRL 将仅依赖冻结 LLM 进行探索。 阶段 B：价值感知选择。 为确定最终上下文 ${\mathcal{M}}_{ctx}(s)$，我们使用一个在探索（相似性）和利用（效用 $Q$）之间平衡的复合分数，从 $\mathcal{C}(s)$ 中选择 top-$k_2$ 项：

$$\begin{array}{rl}\mathrm{score}(s,z_i,e_i)& =(1-\lambda )\cdot \widehat{\mathrm{sim}}(\mathrm{Emb}(s),\mathrm{Emb}(z_i))+\lambda \cdot \widehat{Q_i}.\end{array}$$

where $\hat{\cdot }$ denotes z-score normalization and $\lambda \in [0,1]$ modulates the trade-off. This mechanism filters out ``distractor'' memories—those semantically similar but with low historical utility. As detailed in Section 5.2, normalization and strict similarity thresholds are essential for noise filtering and maintaining stability during self-evolution.

其中 $\hat{\cdot }$ 表示 z-score 归一化，$\lambda \in [0,1]$ 调节权衡。 这一机制过滤掉“干扰”记忆，即那些语义相似但历史效用较低的记忆。 如第 5.2 节所详述，归一化和严格相似性阈值对于噪声过滤以及在自演化期间保持稳定至关重要。

4.3 Non-Parametric RL on Memory

The core of MemRL is the continuous refinement of Q-values based on environmental feedback, enabling the agent to ``remember'' what works. During runtime, MemRL performs learning entirely in memory space. With the retrieved context $m$, the agent samples an action $a$ according to the policy $\pi$ defined in Eq. 1. Executing $a$ then yields an environmental reward $r$ (e.g., execution success or scalar score). For the memories actually injected into the input context ${\mathcal{M}}_{\text{ctx}}(s)$, we update their utilities in triplets with a Monte Carlo style rule, i.e., the Eq. 4, following the runtime learning loop shown in Figure 3. This update drives $Q_{\text{new}}$ toward the empirical expected return of using experience $e_i$ under similar intents. Meanwhile, for each sampled trajectory, we use an LLM to summarize the experience, and write it back into the memory bank as a new triplet $(z,e_{\text{new}},Q_{\text{init}})$, enabling continual expansion of experience while keeping the LLM parameters unchanged.

MemRL 的核心是基于环境反馈持续细化 Q-value，使智能体能够“记住”什么有效。 在运行时，MemRL 完全在记忆空间中执行学习。 给定检索到的上下文 $m$，智能体根据公式 1 中定义的策略 $\pi$ 采样动作 $a$。 执行 $a$ 随后产生环境奖励 $r$（例如执行成功或标量分数）。 对于实际注入输入上下文 ${\mathcal{M}}_{\text{ctx}}(s)$ 的记忆，我们按照图3所示的运行时学习循环，使用公式 4 的 Monte Carlo 风格规则更新三元组中的效用。 这一更新推动 $Q_{\text{new}}$ 接近在相似 intent 下使用经验 $e_i$ 的经验预期回报。 同时，对于每条采样轨迹，我们使用 LLM 总结经验，并将其作为新三元组 $(z,e_{\text{new}},Q_{\text{init}})$ 写回记忆库，使经验能够持续扩展，同时保持 LLM 参数不变。

4.4 Theoretical Stability Analysis

We analyze the stability of MemRL from a reinforcement learning perspective, with full analysis provided in the appendix. We posit two standard assumptions: a frozen inference policy $p_{\mathrm{LLM}}$ and a stationary task distribution. Under these conditions, the learning target $\beta (s,m)=\mathbb{E}[r_t|s,m]$ is well-defined, where the expectation is taken over the stochastic rewards resulting from the Inference Policy $p_{LLM}$ distribution. We prove that utility estimates updated via Eq. 4 are unbiased and variance-bounded. Specifically, as $t\to \mathrm{\infty }$:

我们从强化学习视角分析 MemRL 的稳定性，完整分析见附录。 我们提出两个标准假设：冻结推理策略 $p_{\mathrm{LLM}}$ 和平稳任务分布。 在这些条件下，学习目标 $\beta (s,m)=\mathbb{E}[r_t|s,m]$ 是良定义的，其中期望取自 Inference Policy $p_{LLM}$ 分布产生的随机奖励。 我们证明，通过公式 4 更新的效用估计是无偏且方差有界的。 具体而言，当 $t\to \mathrm{\infty }$ 时：

$$\begin{array}{rl}\underset{t\to \mathrm{\infty }}{lim}\mathbb{E}[Q_t]& =\beta (s,m),\\ \underset{t\to \mathrm{\infty }}{lim sup}\mathrm{Var}(Q_t)& \le \frac{\alpha }{2-\alpha }\mathrm{Var}(r_t\mid s,m).\end{array}$$

Furthermore, we address the challenge of the latent retrieval distribution $Pr(s|m)$ shifting during training by framing MemRL as a Generalized Expectation-Maximization (GEM) process. The system performs coordinate ascent on a global objective: the retrieval ranking acts as the Policy Improvement (E-step), while the utility update acts as the Value Update (M-step). By the monotonic improvement theorem, the system converges to a stationary point where the global memory utility stabilizes:

此外，我们通过将 MemRL 表述为 Generalized Expectation-Maximization（GEM） 过程，处理训练期间潜在检索分布 $Pr(s|m)$ 发生漂移的挑战。 系统在一个全局目标上执行坐标上升：检索排序充当 Policy Improvement（E-step），而效用更新充当 Value Update（M-step）。 根据单调改进定理，系统会收敛到全局记忆效用稳定的平稳点：

$$\begin{array}{r}\underset{t\to \mathrm{\infty }}{lim}\mathbb{E}[Q_t(m)]=\sum _{s\in \mathcal{S}(m)}\mathbb{E}[r\mid s,m]Pr(s\mid m).\end{array}$$

where $\mathcal{S}(m)$ is the effective support set of the memory. This formulation guarantees global stability and prevents catastrophic forgetting. Details can be found in the appendix.

其中 $\mathcal{S}(m)$ 是该记忆的有效支撑集。 这一形式化保证了全局稳定性并防止灾难性遗忘。 细节见附录。

5. Experiments

5.1 Experimental Setup

Baselines & Benchmarks. We compare MemRL against RAG-based (RAG, Self-RAG), Agentic Memory (Mem0, MemP), and Test-Time Scaling (Pass@$k$) baselines under a frozen-backbone setting. Evaluations span four domains: BigCodeBench (coding), ALFWorld (navigation), LifelongAgent Bench (OS/DB), and Humanity's Last Exam(HLE). Details are in Appendix. Backbones are selected per benchmark to avoid no-signal or ceiling problems, ensuring valid learning signals, while Appendix provides a unified comparison to demonstrate cross-task consistency under identical capacity.

基线与基准。 我们在冻结 backbone 设置下，将 MemRL 与基于 RAG 的基线（RAG、Self-RAG）、智能体记忆基线（Mem0、MemP）和测试时扩展基线（Pass@$k$）进行比较。 评估覆盖四个领域：BigCodeBench（代码）、ALFWorld（导航）、LifelongAgent Bench（OS/DB）和 Humanity's Last Exam（HLE）。 细节见附录。 我们按基准选择 backbone，以避免无信号或天花板问题，确保有效学习信号；同时，附录提供了统一比较，以展示相同能力下的跨任务一致性。

Metrics. We employ two metrics: (1) Success Rate (SR), the ratio of tasks completed in an epoch; (2) Cumulative Success Rate (CSR), the proportion of tasks solved at least once across epochs. We evaluate our MemRL and baselines under two distinct settings: Runtime Learning, which assesses the ability to learn and adapt within a training session, and Transferring, which evaluates the generalization capability of the learned memory on unseen tasks. Implementation and reproducibility details, including all prompts used in our experiments, can be found in Appendix.

指标。 我们使用两个指标：(1) Success Rate（SR），即一个 epoch 中完成任务的比例；(2) Cumulative Success Rate（CSR），即跨 epoch 至少解决过一次的任务比例。 我们在两个不同设置下评估 MemRL 和基线：Runtime Learning 评估训练会话内学习和适应的能力，Transferring 评估学得记忆在未见任务上的泛化能力。 实现和可复现性细节，包括实验中使用的所有提示，见附录。

表1：Runtime Learning results. We compare MemRL against various baselines over 10 epochs. The results are reported as Last Epoch Success Rate / Cumulative Success Rate (CSR). The Average column indicates the mean performance across all benchmarks.

	BigCodeBench	Lifelong Agent Bench		ALFWorld	HLE	Average
Method	Code Gen (Last / CSR)	OS Task (Last / CSR)	DB Task (Last / CSR)	Exploration (Last / CSR)	Knowledge Frontier (Last / CSR)	(Last / CSR)
Model	GPT-4o	GPT-4o-mini	GPT-4o-mini	GPT-5-mini	Gemini-3-pro	-
No Memory	0.485	0.674	0.860	0.777	0.357	0.631
Pass@10	-- / 0.577	-- / 0.756	-- / 0.928	-- / 0.928	-- / 0.524	-- / 0.743
RAG	0.475 / 0.483	0.690 / 0.700	0.914 / 0.916	0.887 / 0.930	0.430 / 0.475	0.679 / 0.699
Self-RAG	0.497 / 0.561	0.646 / 0.732	0.891 / 0.898	0.907 / 0.962	0.423 / 0.475	0.673 / 0.726
Mem0	0.487 / 0.495	0.670 / 0.702	0.920 / 0.926	0.894 / 0.969	0.436 / 0.470	0.681 / 0.712
MemP	0.578 / 0.602	0.736 / 0.742	0.960 / 0.966	0.885 / 0.919	0.522 / 0.570	0.736 / 0.760
MemRL (ours)	0.595 / 0.627	0.788 / 0.804	0.960 / 0.972	0.949 / 0.981	0.570 / 0.606	0.772 / 0.798

表2：Transfer Learning results on BigCodeBench, Lifelong Agent Bench and ALFWorld. We compare MemRL against various baselines using the best validation results. The Average column represents the mean Success Rate across all benchmarks.

	BigCodeBench	Lifelong Agent Bench		ALFWorld	Average
Method	Code Generation (Success Rate)	OS Task (Success Rate)	DB Task (Success Rate)	Exploration (Success Rate)	(Success Rate)
Model	GPT-4o	GPT-4o-mini	GPT-4o-mini	GPT-5-mini	-
No Memory	0.485	0.673	0.841	0.836	0.709
RAG	0.479	0.713	0.920	0.950	0.765
Self-RAG	0.500	0.653	0.881	0.950	0.746
Mem0	0.485	0.686	0.935	0.950	0.764
MemP	0.494	0.720	0.928	0.921	0.766
MemRL (ours)	0.508	0.746	0.942	0.979	0.794

5.2 Main Results

Runtime Learning Results. As detailed in Table 1, MemRL demonstrates robust superiority across all domains, surpassing the strongest baseline (MemP) by an average of $+3.8\mathrm{\%}$ in Cumulative Success Rate (CSR). The gains are most significant in exploration-intensive environments like ALFWorld and OS tasks (both $+6.2\mathrm{\%}$), while maintaining a steady lead on the challenging HLE benchmark ($+3.6\mathrm{\%}$). This confirms that our value-based mechanism, unlike MemP's heuristic retrieval, effectively filters noise to retain high-utility procedural patterns.

运行时学习结果。 如表1所详述，MemRL 在所有领域都表现出稳健优势，在 Cumulative Success Rate（CSR）上平均超过最强基线 MemP $+3.8\mathrm{\%}$。 在 ALFWorld 和 OS 任务等探索密集型环境中，增益最显著（均为 $+6.2\mathrm{\%}$），同时在具有挑战性的 HLE 基准上保持稳定领先（$+3.6\mathrm{\%}$）。 这证实，与 MemP 的启发式检索不同，我们的基于价值的机制能够有效过滤噪声，以保留高效用程序性模式。

Transferring Results. We evaluate memory transferability by freezing the memory bank after training and testing on held-out sets. As shown in Table 2, MemRL exhibits superior transferability, outperforming the strongest baseline (MemP) by an average of $+2.8\mathrm{\%}$ in Success Rate. The advantage is particularly pronounced in complex environments like ALFWorld ($+5.8\mathrm{\%}$) and OS tasks ($+2.6\mathrm{\%}$). These margins validate that our Two-Phase Retrieval effectively filters low-value noise, retaining high-utility procedural patterns that generalize robustly to unseen scenarios.

迁移结果。 我们通过在训练后冻结记忆库并在留出集合上测试来评估记忆可迁移性。 如表2所示，MemRL 展现出更强的可迁移性，在 Success Rate 上平均超过最强基线 MemP $+2.8\mathrm{\%}$。 这一优势在 ALFWorld（$+5.8\mathrm{\%}$）和 OS 任务（$+2.6\mathrm{\%}$）等复杂环境中尤其明显。 这些差距验证了我们的 Two-Phase Retrieval 能够有效过滤低价值噪声，保留可稳健泛化到未见场景的高效用程序性模式。

5.3 Ablations

图4：OS Interaction Performance. MemRL 与基线（MemP、RAG）的性能比较。实线表示 Epoch Success Rate，虚线表示 Cumulative Success Rate（CSR）。MemRL 展示出更强稳定性和不断扩大的 CSR 性能差距。

Effectiveness of Runtime RL. To isolate the efficacy of runtime RL, we compare MemRL and its RAG-based variant against their non-RL counterparts (MemP and standard RAG) in the OS interaction environment. As shown in Figure 4, while initial performance is comparable, a clear divergence emerges as training progresses: MemRL achieves a smoother learning curve and superior stability. Crucially, this advantage is most pronounced in the Cumulative Success Rate (dashed lines), where the monotonic widening gap indicates that the RL-driven value function effectively filters noisy memories and consolidates successful experiences.

运行时 RL 的有效性。 为隔离运行时 RL 的效果，我们在 OS 交互环境中将 MemRL 及其基于 RAG 的变体与不带 RL 的对应方法（MemP 和标准 RAG）进行比较。 如图4所示，虽然初始表现相近，但随着训练推进出现明显分化：MemRL 获得更平滑的学习曲线和更强稳定性。 关键的是，这一优势在 Cumulative Success Rate（虚线）中最明显，单调扩大的差距表明 RL 驱动的价值函数能够有效过滤噪声记忆并巩固成功经验。

图5：Q-value weighting factor λ 的消融。比较 λ ∈ {0, 0.25, 0.5, 0.75, 1} 下的表现。实线表示 Epoch Success Rate，虚线表示 Cumulative Success Rate（CSR）。平衡设置 λ = 0.5 在相关性与有用性之间取得最优权衡。

Impact of Q-Value Weighting. To determine the optimal equilibrium between semantic grounding and value-based exploitation, we evaluate the Q-weighting factor $\lambda \in \{0,0.25,0.5,0.75,1\}$. As shown in Figure 5, performance exhibits a clear concave trend peaking at $\lambda =0.5$. Deviating toward extremes degrades results: pure semantic retrieval ($\lambda =0$) plateaus due to an inability to filter functional distractors, while excessive RL weight ($\lambda \to 1$) induces volatility and context detachment. This confirms that $\lambda =0.5$ represents an effective balance, where semantic similarity guarantees content relevance and Q-value ensures its helpfulness.

Q-value 权重的影响。 为确定语义锚定与基于价值的利用之间的最优平衡，我们评估 Q-weighting factor $\lambda \in \{0,0.25,0.5,0.75,1\}$。 如图5所示，性能呈现清晰的凹形趋势，并在 $\lambda =0.5$ 处达到峰值。 偏向极端会降低结果：纯语义检索（$\lambda =0$）由于无法过滤功能性干扰项而停滞，而过高的 RL 权重（$\lambda \to 1$）会引发波动和上下文脱离。 这证实 $\lambda =0.5$ 是一种有效平衡，其中语义相似性保证内容相关性，Q-value 保证其有用性。

表3：Ablation on Retrieval Scope. We compare MemRL against an ablated version restricted to Single-Task Reflection.

Setting	BCB	OS	DB	ALFWorld	HLE	Avg.
Single-Task Reflection	0.614	0.714	0.938	0.930	0.610	0.761
MemRL (Cross-Task)	0.627	0.804	0.972	0.981	0.606	0.798

Ablation Analysis: Cross-Task vs. Single-Task Optimization. To investigate the source of our performance gains, we conduct an ablation study in Table 3 by restricting the memory retrieval scope. We compare the full MemRL (Cross-Task Retrieval) against an ablated setting that only utilizes feedback from the single task instance, conceptually equivalent to Reflexion. MemRL demonstrates superior performance in structured environments, particularly on OS-Agent ($+9.0\mathrm{\%}$) and ALFWorld ($+5.1\mathrm{\%}$). These benchmarks exhibit high intra-dataset similarity, allowing MemRL to effectively perform horizontal transfer—retrieving and adapting successful policies from semantically similar historical tasks. While on the HLE benchmark, the single-task baseline ($0.610$) is tied with MemRL ($0.606$). We attribute this to the HLE dataset's low internal semantic similarity ($0.186$), as detailed in Appendix. This prevents effective cross-task generalization, forcing the agent to rely solely on single feedback.

消融分析：跨任务优化与单任务优化。 为研究性能增益的来源，我们在表3中通过限制记忆检索范围进行消融研究。 我们将完整 MemRL（Cross-Task Retrieval）与一种只利用单个任务实例反馈的消融设置比较，后者在概念上等价于 Reflexion。 MemRL 在结构化环境中表现更好，尤其是在 OS-Agent（$+9.0\mathrm{\%}$）和 ALFWorld（$+5.1\mathrm{\%}$）上。 这些基准表现出较高的数据集内相似性，使 MemRL 能够有效执行横向迁移，即从语义相似的历史任务中检索并适配成功策略。 而在 HLE 基准上，单任务基线（$0.610$）与 MemRL（$0.606$）持平。 我们将其归因于 HLE 数据集较低的内部语义相似性（$0.186$），详见附录。 这会阻止有效的跨任务泛化，迫使智能体仅依赖单次反馈。

图6：Retrieval Size（k1, k2）的消融。在 HLE（CS/AI）子集上比较稀疏（3/1）、适中（5/3）和密集（10/5）检索设置。适中设置取得最优权衡。

Sensitivity to Retrieval Size ($k_1$ and $k_2$). To investigate the impact of retrieval bandwidth, we compare three memory density configurations on the HLE (CS/AI) subset benchmark: sparse ($k_1=3,k_2=1$), moderate ($k_1=5,k_2=3$), and dense ($k_1=10,k_2=5$). As shown in Figure 6, performance follows an inverted-U trajectory, illustrating the trade-off between information sufficiency and context noise. The sparse setting limits performance due to insufficient guidance, whereas the dense setting degrades success rate by introducing distractions into the reasoning context. Consequently, the moderate configuration ($k_1=5,k_2=3$) achieves the best result, effectively maximizing the signal-to-noise ratio of the retrieved context.

对 Retrieval Size（$k_1$ 和 $k_2$）的敏感性。 为研究检索带宽的影响，我们在 HLE（CS/AI）子集基准上比较三种记忆密度配置：稀疏（$k_1=3,k_2=1$）、适中（$k_1=5,k_2=3$）和密集（$k_1=10,k_2=5$）。 如图6所示，性能遵循倒 U 型轨迹，说明了信息充足性与上下文噪声之间的权衡。 稀疏设置由于指导不足而限制性能，而密集设置则因向推理上下文引入干扰而降低成功率。 因此，适中配置（$k_1=5,k_2=3$）取得最佳结果，有效最大化了检索上下文的信噪比。

5.4 Discussion

In this section, we delve deeper into the mechanisms driving MemRL's performance, connecting empirical results to the challenge of balancing knowledge retention and adaptation.

在本节中，我们进一步深入探讨驱动 MemRL 性能的机制，并将实证结果与平衡知识保留和适应这一挑战联系起来。

图7：Q-Value Analysis。（a）Pearson r = 0.861 证实 Critic 的预测能力。（b）高 Q 区间中的失败记忆（约 12%）表明潜在策略效用。

Predictive Power of the Q Critic. As shown in Figure 7, the learned Q-values exhibit a strong positive correlation (Pearson $r=0.861$) with empirical task success rates, rising from $21.5\mathrm{\%}$ in the lowest-confidence bin to $88.1\mathrm{\%}$ in the highest, which confirms the Critic's ability to effectively rank memories by success likelihood. Beyond simple ranking, memory composition analysis (Figure 7) reveals that the agent retains a small fraction of ``failure'' memories ($\sim 12\mathrm{\%}$) even in high-Q bins ($0.9-1.0$), suggesting that Q-values capture utility beyond binary outcomes by recognizing strategically useful near-misses. We further substantiate this with concrete case studies in Appendix. This indicates that the Critic prioritizes reusable guidance---including transferable procedural lessons from high-utility failures---rather than merely separating success from failure, thereby offering greater robustness than simple success-replay mechanisms.

Q Critic 的预测能力。 如图7所示，学得的 Q-value 与经验任务成功率呈强正相关（Pearson $r=0.861$），从最低置信区间的 $21.5\mathrm{\%}$ 上升到最高区间的 $88.1\mathrm{\%}$，这证实 Critic 能够按成功可能性有效排序记忆。 除简单排序之外，记忆组成分析（图7）显示，即使在高 Q 区间（$0.9-1.0$），智能体也保留了一小部分“失败”记忆（$\sim 12\mathrm{\%}$），这表明 Q-value 能够通过识别策略上有用的近失误来捕获超越二元结果的效用。 我们还通过附录中的具体案例研究进一步证实这一点。 这表明 Critic 优先考虑可复用指导，包括来自高效用失败的可迁移程序性经验，而不只是区分成功与失败，因此比简单的成功回放机制更稳健。

图8：HLE 中 MemRL 和 MemP 的 Epoch Success Rate 与 Cumulative Success Rate（CSR）。

Stability of MemRL. We analyze MemRL through the lens of the stability-plasticity dilemma. The superior CSR (Table 1) confirms that MemRL effectively expands the solution space, enabling the agent to break through local optima. Furthermore, long-term dynamics (Figure 8) reveal a critical stability advantage: while heuristic methods like MemP suffer from catastrophic forgetting—evidenced by a widening gap between CSR and current Success Rate—MemRL maintains synchronized growth. This is theoretically guaranteed by our stability analysis in Section 4.4, which constrains the policy to improve monotonically without drift.

MemRL 的稳定性。 我们从稳定性-可塑性困境视角分析 MemRL。 更高的 CSR（表1）证实 MemRL 有效扩展了解空间，使智能体能够突破局部最优。 此外，长期动态（图8）揭示出关键稳定性优势：MemP 等启发式方法会遭受灾难性遗忘，这表现为 CSR 与当前 Success Rate 之间的差距扩大，而 MemRL 保持同步增长。 这由我们在第 4.4 节中的稳定性分析提供理论保证，该分析约束策略在不漂移的情况下单调改进。

We quantitatively validate these insights using the Forgetting Rate, defined as $FR=N_{\text{lost}}/N_{\text{fail}}$, where $N_{\text{lost}}$ denotes tasks transitioning from previous success to current failure and $N_{\text{fail}}$ is the total number of failures in the current epoch. MemRL achieves the lowest mean forgetting rate ($0.041$), outperforming MemP ($0.051$). Additionally, ablation results demonstrate that removing z-score normalization and similarity gating causes the rate to spike to $0.073$. This confirms that strict filtering is essential to manage utility variance and ensure that self-evolution remains stable.

我们使用 Forgetting Rate 对这些洞见进行定量验证，该指标定义为 $FR=N_{\text{lost}}/N_{\text{fail}}$，其中 $N_{\text{lost}}$ 表示从先前成功转为当前失败的任务，$N_{\text{fail}}$ 是当前 epoch 中的失败总数。 MemRL 取得最低平均遗忘率（$0.041$），优于 MemP（$0.051$）。 此外，消融结果表明，移除 z-score 归一化和相似性门控会使该比率飙升到 $0.073$。 这证实，严格过滤对于管理效用方差并确保自演化保持稳定至关重要。

Extended Analysis. We conduct further investigations to characterize the underlying mechanisms and generalization of MemRL. Specifically, Appendix analyzes MemRL's role as a structural trajectory verifier and the correlation between task similarity and performance gains. Additionally, evaluations of advanced capabilities---including cross-model memory transferability and modular multi-task merging---are detailed in Appendix, demonstrating MemRL's versatility and its capacity for modular capability expansion. We also analyze the cost and efficiency of MemRL in Appendix.

扩展分析。 我们进行了进一步研究，以刻画 MemRL 的底层机制和泛化能力。 具体而言，附录分析了 MemRL 作为结构化轨迹验证器的作用，以及任务相似性与性能增益之间的相关性。 此外，附录还详细评估了高级能力，包括跨模型记忆可迁移性和模块化多任务合并，展示 MemRL 的多功能性及其模块化能力扩展能力。 我们还在附录中分析了 MemRL 的成本和效率。

6. Limitations and Conclusion

Limitations and Future Work. While MemRL establishes a foundation for non-parametric evolution, its runtime dynamics reveal several promising avenues. (i) The current step-wise update, though fast, may introduce high-variance noise in long-horizon trajectories, inspiring us to explore multi-step updates or periodic memory consolidation. (ii) Credit-assignment ambiguity during utility updates, especially with multiple referenced experiences, raises the need for more precise attribution methods like Shapley methods or value decomposition in multi-agent reinforcement learning. (iii) While MemRL improves with increasing task exposure, performance may drift toward reflection-like behavior when task similarity is low, highlighting the need for a sufficiently diverse yet relevant experience base; for industrial deployment, ensuring high task density and hierarchical abstraction may be crucial. Further detailed discussions on these and other challenges, including memory security, dedicated domains, and multi-agent memory sharing, are provided in Appendix.

局限与未来工作。 虽然 MemRL 为非参数演化奠定了基础，但它的运行时动态也揭示出若干有前景的方向。 (i) 当前逐步更新虽然快速，但可能在长程轨迹中引入高方差噪声，这启发我们探索多步更新或周期性记忆整合。 (ii) 效用更新期间的信用分配歧义，特别是在引用多个经验时，会产生对 Shapley 方法或多智能体强化学习中价值分解等更精确归因方法的需求。 (iii) 虽然 MemRL 会随着任务暴露增加而改进，但当任务相似性较低时，性能可能漂移到类似反思的行为，这凸显出需要足够多样且相关的经验库；对于工业部署，确保高任务密度和层级抽象可能很关键。 关于这些和其他挑战的进一步详细讨论，包括记忆安全、专门领域和多智能体记忆共享，见附录。

Conclusion. We proposed MemRL, a non-parametric approach reconciling the stability-plasticity dilemma by treating memory retrieval as a value-based decision process. Through the Intent-Experience-Utility triplet structure and Monte Carlo style updates, MemRL enables agents to self-evolve and differentiate high-utility strategies from semantic noise without weight updates. Extensive evaluations confirm MemRL significantly outperforms baselines in both runtime adaptation and generalization. In a future where static training data becomes scarce, the interactive experiences generated by agents throughout their life cycle will become a new, vital source of knowledge. We hope this paradigm paves the way for building stable, continuously learning agents that efficiently adapt from interaction.

结论。 我们提出 MemRL，这是一种通过将记忆检索视为基于价值的决策过程来调和稳定性-可塑性困境的非参数方法。 通过 Intent-Experience-Utility 三元组结构和 Monte Carlo 风格更新，MemRL 使智能体能够自演化，并在不更新权重的情况下将高效用策略与语义噪声区分开。 大量评估证实，MemRL 在运行时适应和泛化方面都显著优于基线。 在静态训练数据变得稀缺的未来，智能体在整个生命周期中生成的交互经验将成为新的重要知识来源。 我们希望这一范式为构建稳定、持续学习且能从交互中高效适应的智能体铺平道路。

Impact Statement

This paper presents MemRL, a value-reinforced memory retrieval mechanism designed to enhance the long-term reasoning capabilities of LLM Agents. From a broader perspective, our work contributes to the development of more efficient and reliable autonomous systems. By optimizing the memory retrieval process, MemRL reduces the computational overhead of large-scale agent deployments, potentially lowering the environmental impact of AI infrastructure. Furthermore, as LLM agents become more integrated into daily workflows, research into robust memory mechanisms helps ensure these systems remain grounded and consistent in their actions. We do not foresee any immediate negative social consequences specific to this algorithmic advancement, though we acknowledge that all autonomous systems should be deployed with appropriate human oversight to mitigate broader risks associated with AI decision-making.

本文提出 MemRL，这是一种价值强化的记忆检索机制，旨在增强 LLM 智能体的长期推理能力。 从更广阔的视角看，我们的工作有助于发展更高效、更可靠的自主系统。 通过优化记忆检索过程，MemRL 降低了大规模智能体部署的计算开销，并可能降低 AI 基础设施的环境影响。 此外，随着 LLM 智能体更深地融入日常工作流，对稳健记忆机制的研究有助于确保这些系统在行动中保持扎实和一致。 我们没有预见到这种算法进步会带来任何特定的直接负面社会后果，不过我们承认，所有自主系统都应在适当的人类监督下部署，以减轻与 AI 决策相关的更广泛风险。