---

From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory

MemoryAgentRLICML 2026CCF-A北京大学中南大学清华大学智谱 AI华东师范大学中国人民大学

Cai Y, Guo X, Huang X, et al. From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory. ICML, 2026.

https://openreview.net/forum?id=gNWNtstp3r

https://github.com/walkeralan123/MemoPilot

---

从玩家到大师：通过记忆强化学习增强 LLM 智能体的测试时学习

Abstract

Large language model (LLM) agents are increasingly deployed in long-running settings where improving through experience at test time becomes important. A common approach is to update an explicit memory after each interaction to guide future decisions. However, most existing methods rely on hand-designed prompting rules, making it difficult to align memory updates with downstream objectives over multi-step horizons consistently. We propose MemoPilot, a plug-in memory copilot that explicitly trains the memory update process to improve a frozen LLM's performance across sequential interactions. We formulate memory updating as a multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Our training recipe introduces (i) a turn-wise reward signal and (ii) a context-independent, turn-level advantage estimation across rollouts, enabling finer-grained credit assignment and more stable training in multi-turn settings. We evaluate MemoPilot on two testbeds: multi-round Rock--Paper--Scissors (RPS) and Limit Texas Hold'em (LHE). Across both environments, MemoPilot substantially improves test-time learning of a frozen player over strong baselines, ranking first in Elo ratings on both games (1762 on LHE and 1590 on RPS) and outperforming all baseline memory methods and proprietary models, including DeepSeek-V3.2. Our code is publicly available here.

大语言模型（LLM）智能体正越来越多地部署在长期运行的环境中，在测试时通过经验不断改进因而变得十分重要。 一种常见方法是在每次交互后更新显式记忆，以指导未来决策。 然而，现有方法大多依赖手工设计的提示规则，因此难以在多步时间跨度内持续使记忆更新与下游目标保持一致。 我们提出 MemoPilot，这是一种即插即用的记忆副驾驶，它会显式训练记忆更新过程，以提升冻结 LLM 在连续交互中的表现。 我们把记忆更新形式化为多轮决策问题，并使用多轮 GRPO 进行端到端优化。 我们的训练方案引入 (i) 逐轮奖励信号，以及 (ii) 跨轨迹、与上下文无关的轮次级优势估计，从而在多轮场景中实现更细粒度的信用分配和更稳定的训练。 我们在两个测试平台上评估 MemoPilot：多轮石头剪刀布（RPS）和有限注德州扑克（LHE）。 在两个环境中，MemoPilot 都显著增强了冻结玩家的测试时学习能力，在两种游戏的 Elo 排名中均位列第一（LHE 为 1762，RPS 为 1590），并优于所有基线记忆方法和包括 DeepSeek-V3.2 在内的闭源模型。 我们的代码已公开。

图1：MemoPilot 与基线记忆模型在有限注德州扑克连续游戏中的测试时学习动态，使用两个冻结玩家进行评估。累计表现表示截至当前轮次的逐局分数运行平均值。左：记忆训练时使用的玩家 Qwen2.5-14B-Instruct；右：零样本泛化到更强的冻结玩家 Qwen3-235B-A22B。

图2：MemoPilot 框架概览。可训练记忆模型 $G_{\theta }$ 根据交互轨迹迭代更新记忆状态 $m_t$，并把它作为指导提供给即插即用的冻结玩家 $\pi$。所有跨局学习均依赖持续演化的记忆，而 $\pi$ 始终保持不变。

1. Introduction

Large language model (LLM) agents are increasingly used in settings that involve repeated interactions with related tasks, users, or environments. In such settings, a key capability is test-time learning (TTL), where an agent improves over a sequence of interactions by leveraging experience accumulated during deployment. Recent benchmarks and analyses have begun to systematically evaluate such learning capability and efficiency in LLMs and agents, highlighting that the ability to leverage experience can be a central bottleneck for real-world agent reliability and efficiency. This motivates memory-aware agent systems that can accumulate and exploit experience online to improve future decisions.

大语言模型（LLM）智能体正越来越多地用于需要与相关任务、用户或环境反复交互的场景。 在这类场景中，一项关键能力是测试时学习（TTL），即智能体利用部署期间积累的经验，在一系列交互中持续改进。 近期基准和分析开始系统评估 LLM 与智能体的这种学习能力和效率，并指出利用经验的能力可能是真实世界智能体可靠性与效率的核心瓶颈。 这推动了记忆感知智能体系统的发展，使其能够在线积累并利用经验来改进未来决策。

A growing line of work attempts to realize TTL via explicit memory and experience-driven adaptation. Early approaches such as Reflexion and ExpeL demonstrate that agents can iteratively improve by reflecting on interactions and accumulating experience. More recent methods move beyond static storage or naive history reuse and start to incorporate dynamic updates: Dynamic Cheatsheet maintains an evolving memory for test-time adaptation; ReasoningBank distills reusable reasoning strategies from an agent's successes and failures and closes the loop via retrieval and consolidation. Together, these works suggest that dynamic memory update is a promising interface for enabling TTL.

越来越多的工作尝试通过显式记忆和经验驱动的适应来实现 TTL。 Reflexion 和 ExpeL 等早期方法表明，智能体可以通过反思交互并积累经验来迭代改进。 较新的方法超越了静态存储或简单的历史复用，开始引入动态更新：Dynamic Cheatsheet 维护持续演化的记忆以支持测试时适应；ReasoningBank 从智能体的成功与失败中提炼可复用的推理策略，并通过检索与整合形成闭环。 这些工作共同表明，动态记忆更新是实现 TTL 的一种有前景的接口。

However, despite these advances, most existing approaches rely on hand-designed or prompt-based memory update rules, rather than end-to-end optimization of the memory update policy. In our pilot observations, even strong instruction-following LLMs fail to consistently improve across repeated interactions when memory updates are driven only by such heuristic mechanisms, motivating a training signal that directly optimizes memory updates for downstream performance. More broadly, learning to improve at test time has rarely been treated as a trainable capability.

然而，尽管取得了这些进展，现有方法大多仍依赖手工设计或基于提示的记忆更新规则，而非对记忆更新策略进行端到端优化。 我们的初步观察发现，当记忆更新仅由这类启发式机制驱动时，即使是指令遵循能力很强的 LLM，也无法在反复交互中持续改进；这促使我们引入直接针对下游表现优化记忆更新的训练信号。 更广泛地说，在测试时学习改进很少被视为一种可训练能力。

To address this gap, we propose MemoPilot, a plug-in Memory Copilot that explicitly trains the memory update process to improve the performance of a frozen LLM in multi-turn interactions. Inspired by Suzgun et al., we view memory as an evolving artifact that refines across multiple interactions. We treat memory updating as a trainable multi-turn decision problem and optimize it end-to-end with multi-turn GRPO. Concretely, we introduce a turn-wise reward signal and a turn-level advantage estimation across rollouts, which provides finer-grained credit assignment and stabilizes learning in multi-turn settings. This approach yields a natural proxy task where memory quality is assessed by downstream task performance. Multi-turn training is essential as it teaches memory update through an iterative "hypothesize-and-verify" cycle: observes evidence from the current experience, proposes or refines hypotheses, verifies them against accumulated evidence, and corrects prior conclusions.

为弥补这一缺口，我们提出 MemoPilot，这是一种即插即用的记忆副驾驶，它显式训练记忆更新过程，以提升冻结 LLM 在多轮交互中的表现。 受 Suzgun 等人的启发，我们把记忆视为一种在多次交互中不断完善的演化产物。 我们把记忆更新视为可训练的多轮决策问题，并使用多轮 GRPO 进行端到端优化。 具体而言，我们引入逐轮奖励信号和跨轨迹的轮次级优势估计，从而提供更细粒度的信用分配，并稳定多轮场景中的学习。 这种方法自然形成一个代理任务，其中记忆质量由下游任务表现来衡量。 多轮训练至关重要，因为它通过迭代的“假设并验证”循环来教授记忆更新：观察当前经验中的证据，提出或完善假设，根据累积证据验证假设，并纠正先前结论。

We evaluate MemoPilot on two strategic games including multi-round Rock--Paper--Scissors (RPS) and Limit Texas Hold’em (LHE) because they closely match the TTL setting and satisfy three desiderata: (i) learnability under cross-game interaction: there exists exploitable, opponent-specific behavioral structure that can be discovered from multi-game experience; (ii) controllability: opponents can be specified by explicit strategies, enabling reproducible interactions and systematic coverage/generalization tests; and (iii) challenge with measurable reward: both environments provide clear outcome rewards suitable for end-to-end optimization, yet require non-trivial adaptation. LHE introduces imperfect information and rich hand-level variation that acts as natural probes of opponent behavior. While RPS has a small action space, multi-round interaction induces history-dependent dynamics; by designing diverse rule-based and mixed-strategy opponents, it remains challenging while allowing scalable and controlled construction of opponent families. Across both testbeds, we show that plugging MemoPilot into a frozen player substantially improves test-time learning performance over strong baselines.

我们在两种战略游戏上评估 MemoPilot，包括多轮石头剪刀布（RPS）和有限注德州扑克（LHE），因为它们与 TTL 场景高度吻合，并满足三个要求：(i) 跨局交互下的可学习性：存在可从多局经验中发现、可被利用且与对手相关的行为结构；(ii) 可控性：可以用明确策略指定对手，从而实现可复现交互以及系统性的覆盖与泛化测试；(iii) 具有可测奖励的挑战性：两个环境都提供适合端到端优化的明确结果奖励，同时又需要非平凡的适应。 LHE 引入不完全信息和丰富的手牌级变化，可作为探测对手行为的自然手段。 RPS 的动作空间虽然较小，但多轮交互会产生依赖历史的动态；通过设计多样的规则型和混合策略对手，它仍具有挑战性，同时支持以可扩展、可控的方式构造对手族。 在两个测试平台上，我们都表明，把 MemoPilot 接入冻结玩家后，其测试时学习表现显著优于强基线。

Our main contributions are:

1. We propose MemoPilot, a plug-in memory pilot that improves a frozen LLM player’s test-time learning behavior across repeated interactions by training the memory update process end-to-end.

2. We introduce a multi-turn GRPO training recipe for memory updating with turn-wise rewards and turn-level advantage estimation, enabling stable credit assignment in multi-turn test-time learning rollouts.

3. We validate MemoPilot on controlled game testbeds, demonstrating consistent gains in test-time learning.

我们的主要贡献如下：

1. 我们提出 MemoPilot，这是一种即插即用的记忆副驾驶，通过端到端训练记忆更新过程，改善冻结 LLM 玩家在反复交互中的测试时学习行为。

2. 我们提出一种面向记忆更新的多轮 GRPO 训练方案，结合逐轮奖励与轮次级优势估计，在多轮测试时学习轨迹中实现稳定的信用分配。

3. 我们在受控游戏测试平台上验证 MemoPilot，展示了稳定的测试时学习增益。

2. Preliminaries

Test-time learning (TTL) studies settings where an agent receives a stream of related tasks or interactions and improves its performance over time by leveraging experience accumulated during deployment. The stream is revealed sequentially (without access to future interactions), so adaptation must be done online based on past experience.

测试时学习（TTL）研究这样的场景：智能体接收一连串相关任务或交互，并利用部署期间积累的经验逐步提升表现。 这一序列按顺序揭示，智能体无法访问未来交互，因此必须根据过去经验在线完成适应。

In this work, we focus on a sequential-game TTL setting for strategic interactions. Here, each TTL unit is a game (or match) played against an opponent, and the agent is evaluated by the game outcome (e.g., win/loss or chip gain), providing a natural reward signal. Crucially, opponents exhibit exploitable strategy structure, making cross-game adaptation meaningful: information inferred from earlier games can improve decisions in later games.

在这项工作中，我们聚焦战略交互中的连续游戏 TTL 场景。 其中，每个 TTL 单元都是一局与对手进行的游戏（或比赛），智能体根据游戏结果接受评估，例如胜负或筹码收益，从而获得自然的奖励信号。 关键在于，对手表现出可利用的策略结构，因此跨局适应具有意义：从早期游戏推断的信息可以改进后续游戏中的决策。

Notation. We denote the sequence of games by $\{g_t{\}}_{t=1}^T$. Game $g_t$ yields an interaction trajectory $e_t$ and a scalar reward $r_t\in \mathbb{R}$. We consider a memory-based TTL formulation where learning depends on an explicit textual memory updated online without updating model parameters. A memory model $G_{\theta }$ reads accumulated experience and produces a textual memory

符号。 我们用 $\{g_t{\}}_{t=1}^T$ 表示游戏序列。 游戏 $g_t$ 产生交互轨迹 $e_t$ 和标量奖励 $r_t\in \mathbb{R}$。 我们采用基于记忆的 TTL 形式，其中学习依赖在线更新的显式文本记忆，而不更新模型参数。 记忆模型 $G_{\theta }$ 读取累积经验并生成文本记忆：

$$m_t=G_{\theta }(e_t,m_{t-1}),m_0=\mathrm{\varnothing }.$$

which is provided to a fixed player model $\pi$ in subsequent games. The player itself is stateless across games: in each game, it only conditions on the current memory. The interaction evolves as

该记忆会在后续游戏中提供给固定玩家模型 $\pi$。 玩家本身在不同游戏之间是无状态的：每局游戏只以当前记忆为条件。 交互过程演化如下：

$$\begin{array}{rlrl}{e}_1& \sim \pi (\cdot \mid m_0),& m_1& =G_{\theta }(e_1,m_0),\\ e_{t+1}& \sim \pi (\cdot \mid m_t),& m_{t+1}& =G_{\theta }(e_{t+1},m_t).\end{array}$$

3. Method

We now present MemoPilot, a dynamic experiential memory model trained via multi-turn reinforcement learning. Given the sequential-game TTL setup in Section 2, we view memory updating as a sequential decision process, where the generator must learn to extract and express strategic insights that maximize the agent's cumulative performance across an episode of games.

下面介绍 MemoPilot，这是一种通过多轮强化学习训练的动态经验记忆模型。 基于第 2 节的连续游戏 TTL 设置，我们把记忆更新视为连续决策过程，其中生成器必须学会提取并表达战略洞见，以最大化智能体在一段游戏序列中的累积表现。

3.1 Multi-Turn Memory Generation as a Markov Decision Process (MDP)

Following Section 2 and the memory update equation, we cast multi-turn memory updating as a sequential decision problem $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$, where the memory model acts as a policy that must balance information extraction and strategic guidance across multiple interactions.

遵循第 2 节和记忆更新公式，我们把多轮记忆更新表述为连续决策问题 $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R})$，其中记忆模型充当策略，必须在多次交互中平衡信息提取与战略指导。

Formally, the state space $\mathcal{S}$ consists of observation tuples $s_t=(e_t,m_{t-1})$, where $e_t$ is the latest game trajectory and $m_{t-1}$ is the previous memory. The action space $\mathcal{A}$ is the space of textual memories, and the generator samples $m_t\sim G_{\theta }(\cdot \mid s_t)$. We associate each game with an environment instance $E_t$ (e.g., poker private/public cards and positions), sampled from an environment distribution and varying across turns, capturing partial observability. The transition dynamics $\mathcal{P}$ is induced by the frozen player $\pi$ interacting with the opponent under $E_{t+1}$ conditioned on $m_t$, yielding the next trajectory $e_{t+1}$ and scalar reward $r_{t+1}$. The reward function $\mathcal{R}$ returns the observed game outcome $r_t$ for turn $t$ (a task-defined scalar).

形式上，状态空间 $\mathcal{S}$ 由观测元组 $s_t=(e_t,m_{t-1})$ 构成，其中 $e_t$ 是最新游戏轨迹，$m_{t-1}$ 是先前记忆。 动作空间 $\mathcal{A}$ 是文本记忆空间，生成器从 $m_t\sim G_{\theta }(\cdot \mid s_t)$ 中采样。 我们为每局游戏关联一个环境实例 $E_t$，例如扑克中的私有牌、公共牌和位置；它从环境分布中采样并随轮次变化，以刻画部分可观测性。 转移动态 $\mathcal{P}$ 由冻结玩家 $\pi$ 在 $E_{t+1}$ 下、以 $m_t$ 为条件与对手交互而诱导，产生下一条轨迹 $e_{t+1}$ 和标量奖励 $r_{t+1}$。 奖励函数 $\mathcal{R}$ 返回第 $t$ 轮观测到的游戏结果 $r_t$，它是由任务定义的标量。

An episode unfolds as $T$ games with interleaved memory updates following the memory update equation. The player then uses $m_t$ in game $t+1$. Since the first game serves as initial exploration without learned guidance, we define the episode return as the sum of rewards from the memory-guided games:

一个回合由 $T$ 局游戏组成，其间按照记忆更新公式交替进行记忆更新。 随后，玩家在第 $t+1$ 局游戏中使用 $m_t$。 由于第一局游戏充当没有已学习指导的初始探索，我们把回合回报定义为受记忆指导的游戏奖励之和：

$$R(\tau )=\sum _{t=1}^{T-1}{r}_{t+1}.$$

where $\tau$ denotes the full trajectory. The training objective is to maximize expected return over opponent strategies $\sigma$ and trajectories $\tau$ generated by the memory policy:

其中 $\tau$ 表示完整轨迹。 训练目标是在对手策略 $\sigma$ 和由记忆策略生成的轨迹 $\tau$ 上最大化期望回报：

$${\theta }^{\ast }=\arg \underset{\theta }{max}{\mathbb{E}}_{\sigma ,\tau }[R(\tau )].$$

To make optimization practical in multi-turn, stochastic environments, we use a turn-level, low-variance one-step proxy signal for advantage estimation that attributes outcomes to the most recent memory update, improving training stability and sample efficiency.

为了让优化能在多轮随机环境中实际运行，我们使用轮次级、低方差的单步代理信号进行优势估计，把结果归因于最近一次记忆更新，从而提升训练稳定性和样本效率。

图3：用于记忆更新的多轮 GRPO，采用单步（下一局）代理奖励和轮次级优势。每条轨迹 $i$ 产生一系列记忆状态 $\{m_{i,t}\}$；我们为每轮 $t$ 分配单步代理回报 $R_{i,t}=r_{i,t+1}$，计算轮次级组归一化优势，并将其应用于 $m_{i,t}$ 的 token。

3.2 Training with Multi-Turn GRPO

To optimize the objective above, we adopt Group Relative Policy Optimization (GRPO), which has proven effective for training LLM agents in multi-turn settings. In the rollout phase, the policy model $G_{{\theta }_{\mathrm{old}}}$ generates $G$ parallel episode rollouts for each opponent strategy $\sigma$. Each episode $i$ produces $T-1$ memory generations $\{m_{i,1},m_{i,2},\dots ,m_{i,T-1}\}$, where each $m_{i,t}$ decomposes into tokens $(m_{i,t,1},m_{i,t,2},\dots ,m_{i,t,|m_{i,t}|})$. Let $\{R_{i,t}{\}}_{i=1}^G$ denote the per-turn rewards at turn $t$. The group-normalized advantage is:

为了优化上述目标，我们采用组相对策略优化（GRPO），该方法已被证明可有效训练多轮场景中的 LLM 智能体。 在轨迹采样阶段，策略模型 $G_{{\theta }_{\mathrm{old}}}$ 针对每个对手策略 $\sigma$ 生成 $G$ 条并行回合轨迹。 每个回合 $i$ 产生 $T-1$ 次记忆生成 $\{m_{i,1},m_{i,2},\dots ,m_{i,T-1}\}$，其中每个 $m_{i,t}$ 分解为 token $(m_{i,t,1},m_{i,t,2},\dots ,m_{i,t,|m_{i,t}|})$。 令 $\{R_{i,t}{\}}_{i=1}^G$ 表示第 $t$ 轮的逐轮奖励。 组归一化优势为：

$${\hat{A}}_{i,t,k}=R_{i,t}-\mathrm{mean}(\{R_{i,t}{\}}_{i=1}^G),R_{i,t}=r_{i,t+1}.$$

Following Liu et al., we omit standard deviation normalization. This turn-specific advantage is applied to all tokens within the same memory generation step.

遵循 Liu 等人的做法，我们省略标准差归一化。 这一轮次特定优势会应用于同一次记忆生成步骤中的所有 token。

While the objective above optimizes the cumulative episode return, in practice we estimate turn-level advantages using the one-step outcome $R_{i,t}=r_{i,t+1}$. Using long-horizon returns would couple the learning signal to future stochasticity (e.g., different dealt hands), amplifying environment noise and making credit assignment unstable. The one-step proxy avoids this issue and yields a cleaner turn-wise learning signal, improving stability and sample efficiency for context learning.

尽管上述目标优化的是回合累积回报，但在实践中，我们使用单步结果 $R_{i,t}=r_{i,t+1}$ 估计轮次级优势。 使用长程回报会把学习信号与未来随机性耦合，例如不同的发牌结果，从而放大环境噪声并使信用分配不稳定。 单步代理避免了这一问题，并产生更干净的逐轮学习信号，从而提升上下文学习的稳定性和样本效率。

As our approach spans multiple turns, each episode generates $T-1$ context-independent memory updates. Inspired by Yu et al., we optimize each memory generation step, extending the loss from the standard (group, token) structure to (group, turn, token). Let $r_{i,t,k}(\theta )$ denote the importance sampling weight for the $k$-th token of memory $m_{i,t}$:

由于我们的方法跨越多个轮次，每个回合会生成 $T-1$ 次彼此独立的记忆更新。 受 Yu 等人的启发，我们优化每个记忆生成步骤，把损失从标准的 (group, token) 结构扩展为 (group, turn, token)。 令 $r_{i,t,k}(\theta )$ 表示记忆 $m_{i,t}$ 中第 $k$ 个 token 的重要性采样权重：

$$r_{i,t,k}(\theta )=\frac{G_{\theta }(m_{i,t,k}\mid e_{i,t},m_{i,t-1},m_{i,t,<k})}{G_{{\theta }_{\mathrm{old}}}(m_{i,t,k}\mid e_{i,t},m_{i,t-1},m_{i,t,<k})}.$$

The multi-turn GRPO objective with clipped surrogate and token-level averaging is:

采用裁剪代理目标和 token 级平均的多轮 GRPO 目标为：

$$\begin{array}{rl}\mathcal{J}(\theta )=& {\mathbb{E}}_{\begin{array}{c}\sigma \sim \mathcal{S},\\ \{m_{i,t}\}\sim G_{{\theta }_{\mathrm{old}}}\end{array}}[\frac{1}{\sum _{i=1}^G\sum _{t=1}^{T-1}|m_{i,t}|}\sum _{i=1}^G\sum _{t=1}^{T-1}\sum _{k=1}^{|m_{i,t}|}{\mathcal{C}}_{i,t,k}],\\ {\mathcal{C}}_{i,t,k}=& min(r_{i,t,k}(\theta ){\hat{A}}_{i,t,k},\\ & \mathrm{clip}(r_{i,t,k}(\theta ),1-\epsilon ,1+\epsilon ){\hat{A}}_{i,t,k}).\end{array}$$

The complete training procedure is summarized in the appendix.

完整训练流程汇总于附录。

Defining the Memory Space. To support iterative refinement, we structure the memory space into three components: (1) a diagnostic analysis that summarizes the evidence from recent interactions and updates hypotheses about the opponent strategy (Identification); (2) an explicit maintained belief state that records the current hypotheses and their confidence or verification status across turns under a fixed memory budget (Maintenance); and (3) concise, actionable guidance that the frozen player can execute in the next game (Guidance). During inference, these components enable an iterative update process: the generator revises its diagnosis and maintained beliefs as new evidence arrives, and updates the guidance accordingly. In addition, the verification or confidence signal in the maintained state provides a natural stopping criterion: once the hypothesis is sufficiently confirmed, the agent can continue playing without further memory revision. See the appendix for the exact prompt template and a multi-turn qualitative example of how the memory evolves.

定义记忆空间。 为支持迭代完善，我们把记忆空间组织为三个组成部分：(1) 诊断分析，总结近期交互中的证据并更新关于对手策略的假设（识别）；(2) 显式维护的信念状态，在固定记忆预算下记录当前假设以及它们跨轮次的置信度或验证状态（维护）；(3) 冻结玩家可在下一局中执行的简洁、可操作指导（指导）。 在推理期间，这些组成部分支持迭代更新过程：随着新证据到来，生成器修正诊断和已维护的信念，并相应更新指导。 此外，维护状态中的验证或置信信号提供了自然的停止准则：一旦假设得到充分确认，智能体就可以继续游戏，而无需进一步修改记忆。 精确的提示模板以及记忆如何演化的多轮定性示例见附录。

3.3 Opponent Construction

A key design choice in our framework is constructing a diverse yet controllable opponent pool that enables systematic study of test-time learning. We design the opponent pool under three principles. Controllability: we specify each opponent using executable instructions to enable reproducible rollouts for stable RL training and evaluation. Behavioral diversity: for LHE, we vary action-frequency biases, street-specific aggression profiles, and deceptive modes (e.g., check-raise traps), while for RPS we cover open-loop sequences, one-step reactive rules, and multi-step counter-patterns. Mechanism-based train--test separation: held-out strategies preserve strategic intent while shifting triggers, or the phase where information is revealed, which probes whether memory can maintain and revise hypotheses as evidence accumulates.

我们框架中的一个关键设计选择，是构造多样且可控的对手池，以便系统研究测试时学习。 我们依据三个原则设计对手池。 可控性：我们使用可执行指令指定每个对手，从而生成可复现轨迹，支持稳定的强化学习训练与评估。 行为多样性：对于 LHE，我们改变动作频率偏置、特定下注轮次的进攻模式和欺骗模式，例如过牌加注陷阱；对于 RPS，我们覆盖开环序列、单步反应规则和多步反制模式。 基于机制的训练测试划分：留出策略保留战略意图，但改变触发条件或信息揭示阶段，以检验记忆能否随着证据积累维护并修正假设。

Our construction follows a human-in-the-loop pipeline: experienced players write seed strategies, LLM-based rewriting expands and standardizes the set, and manual verification ensures each strategy is coherent and behaviorally stable under our execution settings. Details of opponent construction and verification are provided in the appendix.

我们的构造遵循人在回路流程：经验丰富的玩家编写种子策略，基于 LLM 的改写扩展并规范策略集合，人工验证则确保每种策略在我们的执行设置下都具有一致性和行为稳定性。 对手构造与验证细节见附录。

Elo-Based Difficulty Calibration. We estimate an Elo rating for each opponent strategy via round-robin head-to-head matches to check that train/test pools span a broad difficulty range. Figure 4 shows the resulting Elo distribution for RPS, where train and held-out opponents cover a broad and relatively uniform difficulty range. We additionally include Gemini-3.0-Flash and DeepSeek-V3.2 as reference baselines without access to specific strategies. We provide the corresponding Elo rating distribution for LHE and more implementation details in the appendix.

基于 Elo 的难度校准。 我们通过循环制两两对战估计每种对手策略的 Elo 评分，以检查训练池和测试池是否覆盖广泛的难度范围。 图4给出了 RPS 的 Elo 分布，其中训练对手和留出对手覆盖了广泛且相对均匀的难度范围。 我们还加入无法访问具体策略的 Gemini-3.0-Flash 和 DeepSeek-V3.2 作为参考基线。 LHE 对应的 Elo 评分分布和更多实现细节见附录。

图4：由两两对战估计的 RPS 对手策略 Elo 评分，表明构造的对手池覆盖广泛且相对均匀的难度范围。蓝色和粉色柱分别表示训练对手和留出对手，紫色柱表示 LLM。

表1：战略游戏上的主结果。报告评估运行的 mean@64。对于 Memory w/ MemoPilot，括号中的数值表示相对 Memory w/ Qwen2.5-14B 的绝对提升。

Method	Qwen2.5-14B-Instruct as Player		Qwen3-235B-A22B as Player
	RPS@5	LHE@5	RPS@5	LHE@5
Baseline Methods
No Memory	0.43	-1.36	0.44	-1.46
Full History	0.02	-1.22	0.03	-1.45
Human-Written Counter-Strategy	1.0	1.08	0.57	0.39
Previous Methods
Reflexion	0.61	-1.27	0.53	-0.85
ExpeL	0.03	-0.39	-0.02	-0.58
MemoryBank	0.43	-0.96	0.48	-1.29
AWM	0.64	-1.17	0.67	-1.32
ReasoningBank	0.81	-1.14	0.81	-0.87
Memory-based Methods
Memory w/ Qwen2.5-7B-Instruct	0.36	-0.13	0.19	-0.85
Memory w/ DeepSeek-V3.2	1.64	-0.78	1.46	-0.60
Memory w/ Gemini-3.0-Flash	0.45	-1.26	0.54	1.16
Memory w/ Qwen2.5-14B-Instruct	0.21	-0.23	0.34	-0.29
Memory w/ MemoPilot	3.28 (+3.10)	2.03 (+2.30)	3.27 (+2.90)	1.31 (+1.60)

4. Experiments

4.1 Experimental Setup

Environments. We evaluate two strategic games from TextArena and RLCard. The first is multi-round Rock--Paper--Scissors (RPS): each game contains 6 consecutive rounds, and both players observe the full history of previous rounds before making each decision. The second is Limit Texas Hold'em (LHE): each player has two private cards and chooses from four actions (Fold, Check, Call, Raise), featuring partial observability and stochastic outcomes.

环境。 我们评估来自 TextArena 和 RLCard 的两种战略游戏。 第一种是多轮石头剪刀布（RPS）：每局游戏包含连续 6 轮，双方玩家在每次决策前都能观察此前所有轮次的完整历史。 第二种是有限注德州扑克（LHE）：每位玩家持有两张私有牌，并从四种动作中选择，即弃牌、过牌、跟注和加注；该游戏具有部分可观测性和随机结果。

Metrics. For RPS, we define the per-game score as the difference between the number of rounds won by the player and by the opponent over a 6-round match. We report RPS@k as the average per-game score over $k$ consecutive games. For LHE, a game consists of a duplicate match, in which the players swap positions while sharing the same card deal. This duplicate-match eliminates variance induced by the card and seating order, enabling a fair comparison of players’ strategic strength. We define the per-game chip as the sum of the final chip counts from these two subgames. We report LHE@k as the average per-game chip over $k$ consecutive games.

指标。 对于 RPS，我们把每局分数定义为在一场 6 轮比赛中玩家获胜轮数与对手获胜轮数之差。 我们报告 RPS@k，即连续 $k$ 局游戏的平均逐局分数。 对于 LHE，一局游戏由一场复式比赛构成，双方玩家在共享同一发牌结果的同时交换位置。 这种复式比赛消除了牌面和座次顺序引起的方差，从而可以公平比较玩家的战略实力。 我们把逐局筹码定义为这两个子局最终筹码数之和。 我们报告 LHE@k，即连续 $k$ 局游戏的平均逐局筹码。

Evaluation. Due to the stochasticity of LLM sampling and game dynamics, we report results as mean@64 for both environments, averaging over 64 evaluation runs across strategies. All memory-based methods are evaluated with a fixed memory budget of 512 tokens. For LHE, we evaluate all methods on the same fixed set of card deals shared across evaluation runs to ensure a fair comparison.

评估。 由于 LLM 采样和游戏动态具有随机性，我们在两个环境中均报告 mean@64，即对跨策略的 64 次评估运行取平均。 所有基于记忆的方法都在固定的 512 token 记忆预算下评估。 对于 LHE，我们让所有方法在各次评估运行共享的同一组固定发牌结果上接受评估，以确保公平比较。

Training Details. We use Qwen2.5-14B-Instruct as the base model of MemoPilot. We train a separate memory model for RPS and LHE. During training, we fix Qwen2.5-14B-Instruct as the player model. We also use it as the opponent model in both training and evaluation. Different opponents are constructed by providing different strategy system prompts. For evaluation, we assess the trained memory model's performance by pairing it with different player models. By default, one training rollout contains 3 consecutive games, during which the agent updates cross-game memory between games. For LHE, we use the same seed within each GRPO group so that rollouts share the same cards at the same turn. Computational cost is discussed in the appendix.

训练细节。 我们使用 Qwen2.5-14B-Instruct 作为 MemoPilot 的基础模型。 我们分别为 RPS 和 LHE 训练记忆模型。 训练期间，我们固定 Qwen2.5-14B-Instruct 作为玩家模型。 在训练和评估中，我们也使用它作为对手模型。 不同的对手通过提供不同的策略系统提示来构造。 评估时，我们把训练后的记忆模型与不同玩家模型配对，以衡量其表现。 默认情况下，一条训练轨迹包含连续 3 局游戏，智能体在游戏之间更新跨局记忆。 对于 LHE，我们在每个 GRPO 组内使用相同随机种子，使各条轨迹在同一轮共享相同牌面。 计算成本在附录中讨论。

Baselines. We compare MemoPilot against a set of baselines, including No Memory, Full History, Human-Written Counter-Strategy, Reflexion, ExpeL, MemoryBank, AWM, and ReasoningBank. No Memory plays $k$ independent games per opponent without any cross-game state. Full History provides the full interaction history from previous games as context. Human-Written Counter-Strategy asks experienced human players to write a concrete exploit-oriented action plan based on the opponent's strategy description, aiming for clarity and executability. Other baselines are implemented in our sequential-game setting following their core mechanisms, using DeepSeek-V3.2 as the base model. Refer to implementation details in the appendix.

基线。 我们将 MemoPilot 与一组基线进行比较，包括 No Memory、Full History、Human-Written Counter-Strategy、Reflexion、ExpeL、MemoryBank、AWM 和 ReasoningBank。 No Memory 针对每个对手进行 $k$ 局彼此独立的游戏，不保留任何跨局状态。 Full History 把此前游戏的完整交互历史作为上下文提供。 Human-Written Counter-Strategy 要求经验丰富的人类玩家根据对手的策略描述编写具体、面向利用弱点的行动计划，并以清晰和可执行为目标。 其他基线遵循各自核心机制在我们的连续游戏场景中实现，并使用 DeepSeek-V3.2 作为基础模型。 实现细节见附录。

图5：根据两两对战计算的 RPS 与 LHE 记忆方法 Elo 排名，数值越高越好。

图6：MemoPilot 与基线记忆模型在连续 RPS 游戏中的测试时学习动态，使用两个冻结玩家进行评估。左：记忆训练时使用的 Qwen2.5-14B-Instruct；右：零样本泛化到更强的 Qwen3-235B-A22B。

4.2 Main Results

We evaluate online test-time learning where the agent plays sequential games against an opponent, updating memory after each game. Table 1 summarizes performance on multi-round RPS and LHE. The results highlight three key observations.

我们评估在线测试时学习：智能体与对手连续进行游戏，并在每局后更新记忆。 表1汇总了多轮 RPS 和 LHE 上的表现。 结果突出了三个关键观察。

MemoPilot Delivers Consistent Gains Over Memory-Free and Prompting Baselines. Across both games, MemoPilot achieves the strongest average performance at five rounds. With Qwen2.5-14B as the frozen player, MemoPilot reaches 3.28 on RPS@5 and 2.03 on LHE@5, while No Memory remains at 0.43 and $-1.36$. Prompting-based memory baselines provide only limited improvements on RPS and are generally negative on LHE, indicating that heuristic memory updates do not reliably translate into better decisions in this online setting.

MemoPilot 相比无记忆和提示基线实现稳定增益。 在两种游戏中，MemoPilot 都取得了五轮时最强的平均表现。 以 Qwen2.5-14B 作为冻结玩家时，MemoPilot 在 RPS@5 和 LHE@5 上分别达到 3.28 和 2.03，而 No Memory 仍只有 0.43 和 $-1.36$。 基于提示的记忆基线在 RPS 上只带来有限改进，在 LHE 上通常仍为负值，说明启发式记忆更新在这种在线场景中不能可靠转化为更好的决策。

Elo Rankings Confirm a Consistent Strength Advantage. Figure 5 aggregates head-to-head outcomes into an Elo score for each method in both games. MemoPilot ranks first on both RPS and LHE with scores of 1590 and 1762, respectively, showing a consistent advantage over prompting-based baselines and memory-free play beyond the specific @5 metric in Table 1. Implementation details are provided in the appendix.

Elo 排名证实了稳定的实力优势。 图5把两两对战结果汇总为每种方法在两类游戏中的 Elo 分数。 MemoPilot 在 RPS 和 LHE 上均位列第一，分数分别为 1590 和 1762；除表1中特定的 @5 指标外，它相较基于提示的基线和无记忆玩法也表现出稳定优势。 实现细节见附录。

Naively Longer Histories Can Hurt, Suggesting the Need for Selective Memory. Full History performs poorly on RPS and stays negative on LHE in Table 1. This degradation suggests that simply appending more interaction rounds can introduce noise and dilute the actionable signal needed for the next move. In contrast, MemoPilot compresses experience into a compact memory that preserves the information most relevant for future rounds.

简单延长历史可能有害，说明需要选择性记忆。 在表1中，Full History 在 RPS 上表现不佳，在 LHE 上仍为负值。 这种退化表明，简单追加更多交互轮次会引入噪声，并稀释下一步行动所需的可操作信号。 相比之下，MemoPilot 把经验压缩成紧凑记忆，同时保留与未来轮次最相关的信息。

MemoPilot Improves Rapidly and Generalizes Across Frozen Players. Beyond final-round scores, Figure 1 and Figure 6 show that MemoPilot improves sharply within the first few games and then continues to accumulate higher performance. The same pattern holds across different frozen players. Notably, although we train the memory model with Qwen2.5-14B-Instruct as the frozen player, it successfully assists a substantially stronger player, Qwen3-235B-A22B, achieving 3.27 on RPS@5 and 1.31 on LHE@5. These results suggest that MemoPilot learns a robust memory update behavior that extracts transferable strategic signals from early experience, rather than relying on brittle, model-specific prompting recipes.

MemoPilot 改进迅速，并能跨冻结玩家泛化。 除最终轮次分数外，图1和图6表明，MemoPilot 在最初几局中迅速改进，随后继续积累更高表现。 同一模式在不同冻结玩家上都成立。 值得注意的是，虽然我们使用 Qwen2.5-14B-Instruct 作为冻结玩家训练记忆模型，但它仍能成功辅助强得多的 Qwen3-235B-A22B，在 RPS@5 和 LHE@5 上分别达到 3.27 和 1.31。 这些结果表明，MemoPilot 学到了一种稳健的记忆更新行为，能够从早期经验中提取可迁移的战略信号，而不是依赖脆弱、模型特定的提示方案。

4.3 Real-World Evaluation on StreamBench

To evaluate whether the learned memory update mechanism transfers beyond games, we extend MemoPilot to StreamBench, a benchmark for continuous improvement of language agents. We use Qwen2.5-14B-Instruct as the execution agent. The evaluation contains 32 held-out episodes, each with 5 sequential tasks sampled from the same CoSQL database or DS-1000 Python library. At each turn, the agent receives a new task, executes it, and incorporates environment feedback. We report overall accuracy (pass@4) averaged across all turns. Table 2 shows that full History provides only marginal gains over No Memory, and prompt-based memory updates with DeepSeek-V3.2 or Qwen2.5-14B-Instruct do not improve performance, while MemoPilot achieves the best performance on both tasks.

为了评估学到的记忆更新机制能否迁移到游戏之外，我们把 MemoPilot 扩展到 StreamBench，这是一个用于评估语言智能体持续改进能力的基准。 我们使用 Qwen2.5-14B-Instruct 作为执行智能体。 评估包含 32 个留出回合，每个回合包含从同一个 CoSQL 数据库或 DS-1000 Python 库中采样的 5 个连续任务。 在每一轮，智能体接收新任务、执行任务并吸收环境反馈。 我们报告所有轮次平均后的总体准确率（pass@4）。 表2表明，Full History 相比 No Memory 只带来很小增益；使用 DeepSeek-V3.2 或 Qwen2.5-14B-Instruct 的提示式记忆更新并未改善表现，而 MemoPilot 在两个任务上都取得了最佳表现。

表2：StreamBench 结果。报告所有轮次平均后的总体准确率（pass@4）。

Method	CoSQL	DS-1000
No Memory	69.5	50.0
Full History	70.0	52.5
Memory w/ DeepSeek-V3.2	67.5	50.0
Memory w/ Qwen2.5-14B	66.0	48.8
Memory w/ MemoPilot	73.5	56.3

5. Analysis

表3：使用不同记忆变体时的表现。对于改写条件，我们使用 DeepSeek-V3.2 把 MemoPilot 生成的记忆后编辑为更自然、专业的英语，同时严格保留全部逻辑、数字和策略。

Memory Input	RPS@5	LHE@5
No Memory	0.43	-1.36
Ground-Truth Opponent Strategy	0.75	-0.48
Human-Written Counter-Strategy	1.00	1.08
MemoPilot	3.28	2.07
+Rewrite w/ DeepSeek-V3.2	3.12	1.65

5.1 Learned Memories Act as Executable Guidance

We study why learned memories outperform hand-crafted alternatives by isolating the form of information provided to the frozen player. Table 3 highlights a key gap between semantic correctness and behavioral usefulness. When given the ground-truth opponent strategy description, the player improves over No Memory, increasing RPS@5 from $0.43$ to $0.75$ and LHE@5 from $-1.36$ to $-0.48$. However, the player still struggles to translate correct facts into reliable decisions. While supplying a human-written counter-strategy helps, it remains notably weaker than MemoPilot: it reaches $1.00$ on RPS and $1.08$ on LHE, whereas MemoPilot achieves $3.28$ and $2.07$ under the same setting. This gap is consistent with the role of test-time learning: MemoPilot continually updates memory from game outcomes, refining hypotheses and action rules as evidence accumulates, which enables rapid adaptation over a few games and robustness to situation-dependent variations in play.

我们通过隔离提供给冻结玩家的信息形式，研究学到的记忆为何优于手工替代方案。 表3揭示了语义正确性与行为实用性之间的关键差距。 获得真实对手策略描述后，玩家相比 No Memory 有所改进，RPS@5 从 $0.43$ 提升到 $0.75$，LHE@5 从 $-1.36$ 提升到 $-0.48$。 然而，玩家仍难以把正确事实转化为可靠决策。 提供人类编写的反制策略虽然有所帮助，但仍明显弱于 MemoPilot：它在 RPS 和 LHE 上分别达到 $1.00$ 和 $1.08$，而相同设置下 MemoPilot 达到 $3.28$ 和 $2.07$。 这一差距与测试时学习的作用一致：MemoPilot 根据游戏结果持续更新记忆，随着证据积累完善假设和行动规则，从而能够在少数几局内快速适应，并对依赖具体局面的玩法变化保持稳健。

To further separate the impact of content from surface phrasing, we additionally rewrite MemoPilot's generated memories with DeepSeek-V3.2 into more natural, professional English while strictly preserving all logic, numbers, and strategy. This rewrite retains most of MemoPilot's gains (3.12 on RPS and 1.65 on LHE), and still substantially outperforms ground-truth alternatives, suggesting that the primary benefit comes from learning decision-relevant strategic content that better identifies opponent tendencies and provides effective action guidance. The remaining gap to MemoPilot indicates that the original phrasing and structure can further help the frozen player execute the advice. More implementation details are provided in the appendix.

为了进一步分离内容与表面措辞的影响，我们还使用 DeepSeek-V3.2 把 MemoPilot 生成的记忆改写成更自然、专业的英语，同时严格保留全部逻辑、数字和策略。 这种改写保留了 MemoPilot 的大部分增益（RPS 为 3.12，LHE 为 1.65），并且仍显著优于真实信息替代方案，说明主要收益来自学习与决策相关的战略内容，这些内容能更好地识别对手倾向并提供有效行动指导。 与 MemoPilot 之间剩余的差距表明，原始措辞和结构还能进一步帮助冻结玩家执行建议。 更多实现细节见附录。

表4：LHE@5 上的记忆格式消融。除标为 w/o RL 的变体外，所有可训练记忆变体都使用相同的多轮 GRPO 方案。

Method	LHE@5
No Memory	-1.36
Full History	-1.22
3-tier memory w/o RL	-0.23
Free-form memory w/ RL	1.04
3-tier memory w/ RL	2.03

5.2 Memory Format Ablation

We isolate the effect of the structured memory format by training a free-form scratchpad variant with the same multi-turn GRPO recipe. Table 4 shows that RL training is essential, as the 3-tier memory without RL only slightly improves over Full History. Under the same RL recipe, the free-form variant improves substantially, but the 3-tier format performs better, suggesting that the structure provides a useful inductive bias for maintaining hypotheses and translating them into executable guidance.

我们使用相同的多轮 GRPO 方案训练自由形式的草稿区变体，以隔离结构化记忆格式的影响。 表4表明，强化学习训练至关重要，因为没有强化学习的三层记忆相比 Full History 只有轻微提升。 在相同强化学习方案下，自由形式变体显著改进，但三层格式表现更好，说明这种结构为维护假设并将其转化为可执行指导提供了有用的归纳偏置。

5.3 Multi-Turn Training Enables Long-Horizon Stability

We also study how the training horizon (episode length during multi-turn GRPO training) influences long-horizon behavior. Figure 7 compares MemoPilot trained with $2$-turn versus $5$-turn rollouts for 10 consecutive games. Longer-horizon training yields consistently higher cumulative performance across game rounds, with the gap emerging after the initial exploration games and widening as evidence accumulates. This suggests that training with longer rollouts better teaches the memory model to preserve hypotheses, revise them when contradicted, and provide guidance that remains effective over extended interaction.

我们还研究训练跨度（多轮 GRPO 训练期间的回合长度）如何影响长程行为。 图7比较了分别使用 $2$ 轮和 $5$ 轮轨迹训练的 MemoPilot 在连续 10 局游戏中的表现。 更长跨度的训练在各局中产生持续更高的累积表现；差距在最初的探索局后出现，并随着证据积累而扩大。 这表明，使用更长轨迹训练能更好地教会记忆模型保留假设、在假设受到反驳时修正它们，并提供在较长交互中仍然有效的指导。

图7：训练跨度对测试时学习表现的影响。比较使用较短（2 轮）和较长（5 轮）轨迹训练的记忆模型，并绘制各局的累积表现。更长跨度的训练带来更加稳定、能在持续游戏中累积的增益。

5.4 Cross-Opponent Evaluation

We further evaluate robustness under different opponents by switching opponents mid-stream. Concretely, the player model first plays $5$ consecutive games against opponent A and then plays $5$ games against a different opponent B, while keeping a single evolving memory state throughout. We report metrics only on the last 5 games (games 6--10). Across test opponents, we pair each evaluation opponent B with a distinct warm-up opponent A via a fixed bijection to balance coverage. Table 5 shows that MemoPilot remains effective after the opponent switch, achieving $3.26$ on LHE, indicating that the learned memory updates can revise prior beliefs rather than overfitting to early experience.

我们通过在交互序列中途切换对手，进一步评估方法面对不同对手时的稳健性。 具体而言，玩家模型先与对手 A 连续进行 $5$ 局游戏，随后与另一个对手 B 进行 $5$ 局，同时在整个过程中保留同一个持续演化的记忆状态。 我们只报告最后 5 局，即第 6--10 局的指标。 在各个测试对手上，我们通过固定双射为每个评估对手 B 配对一个不同的热身对手 A，以平衡覆盖范围。 表5表明，切换对手后 MemoPilot 仍然有效，在 LHE 上达到 $3.26$；这说明学到的记忆更新能够修正先前信念，而不是过拟合早期经验。

表5：在整个交互序列中始终使用同一个演化记忆状态的跨对手评估。报告最后 5 局（第 6--10 局）的表现。冷启动时，智能体只与目标对手 B 进行 5 局游戏；热身时，智能体在 B 上评估前先与对手 A（或 B）进行 5 局游戏。

Setting	RPS@5	LHE@5
No Memory (cold-start on B)	0.43	-1.36
Memory w/ MemoPilot
Cold-start on B (5 games only)	3.28	2.03
Warm-up on A, then B	2.56	3.26
Warm-up on B, then B	5.22	3.58

表6：奖励设计比较：累积奖励是未来受记忆指导游戏的奖励总和，而单步奖励只根据紧随其后的下一局结果为每次记忆更新分配信用。

Method	LHE@5
No Memory	-1.36
Cumulative Reward	0.61
One-step Reward	2.03

5.5 Reward Design

We ablate reward design for training the memory model. We compare optimizing a cumulative return over memory-guided games with using a one-step per-turn assignment that credits each memory update $m_t$ by the next-game reward $r_{t+1}$. We find that per-turn one-step rewards are substantially more stable, while cumulative return often collapses after a certain number of training steps. Table 6 shows that the performance of cumulative reward is far below one-step reward, only 0.61 on LHE. We attribute this to variance from future environmental randomness, which pushes the memory model toward generic memories rather than context-sensitive adaptation.

我们对训练记忆模型的奖励设计进行消融。 我们比较两种方案：优化受记忆指导游戏上的累积回报；使用逐轮单步分配，根据下一局奖励 $r_{t+1}$ 为每次记忆更新 $m_t$ 分配信用。 我们发现，逐轮单步奖励明显更加稳定，而累积回报往往会在训练一定步数后崩溃。 表6表明，累积奖励的表现远低于单步奖励，在 LHE 上只有 0.61。 我们把这一现象归因于未来环境随机性带来的方差，它会推动记忆模型生成通用记忆，而非进行上下文敏感的适应。

5.6 Failure Mode Analysis

MemoPilot's main failure mode is a maintenance--refinement tradeoff. The hypothesize-and-verify cycle accumulates evidence to avoid overreacting to noisy individual games, but this conservatism can make memory stale when an opponent deliberately reverses behavior after the agent has committed to a counter-strategy. Table 7 evaluates such non-stationary and adaptive settings. Performance decreases when opponents switch more frequently or when the opponent is also equipped with memory, but MemoPilot remains substantially above the No Memory baseline.

MemoPilot 的主要失败模式是维护与完善之间的权衡。 假设并验证循环通过积累证据来避免对单局噪声反应过度，但如果智能体已经采用某种反制策略后，对手刻意反转行为，这种保守性就可能使记忆过时。 表7评估了这类非平稳和自适应场景。 当对手切换得更频繁，或对手也配备记忆时，表现会下降，但 MemoPilot 仍显著优于 No Memory 基线。

表7：LHE@5 上面对非平稳或自适应对手时的失败模式分析。

Setting	LHE@5
No Memory	-1.36
Same opponent	2.03
Opponent switches every 5 games	1.76
Opponent switches every 2 games	1.21
Opponent with Memory (DeepSeek-V3.2)	1.25

6. Related Work

Memory-Augmented Language Agents. Equipping LLMs with memory has emerged as a key direction for building adaptive agents. Generative Agents introduced memory streams for social simulation. Subsequent work has explored various memory architectures: Agent Workflow Memory extracts reusable workflows from trajectories; A-MEM proposes agentic memory with self-organization; MEM1 learns to synergize memory and reasoning; MemGen generates latent memory for self-evolving agents; and Buffer of Thoughts maintains thought templates for reasoning. Unlike most prior work that focuses on within-task persistence, we study cross-game strategic memory that must evolve across sequential matches, and we train the memory evolve process end-to-end to optimize downstream utility.

记忆增强语言智能体。 为 LLM 配备记忆已成为构建自适应智能体的一个关键方向。 Generative Agents 引入了用于社会模拟的记忆流。 后续工作探索了多种记忆架构：Agent Workflow Memory 从轨迹中提取可复用工作流；A-MEM 提出具有自组织能力的智能体记忆；MEM1 学习协同记忆与推理；MemGen 为自演化智能体生成潜在记忆；Buffer of Thoughts 则维护用于推理的思维模板。 不同于大多数聚焦任务内持续性的已有工作，我们研究必须在连续比赛中演化的跨局战略记忆，并端到端训练记忆演化过程，以优化下游效用。

Experience-Driven and Lifelong Learning. Recent work has explored how agents can learn from accumulated experience. Reflexion uses verbal self-reflection for improvement, while ExpeL accumulates insights across tasks. Dynamic Cheatsheet maintains evolving memory through heuristic updates; ReasoningBank scales memory through trajectory comparison. SkillWeaver and PolySkill study reusable skills for self-improving or continual agents. These works are complementary to our setting: they largely rely on heuristic or prompt-based experience updates, whereas MemoPilot optimizes the memory update policy directly with downstream reward. Benchmarks including EvaLearn, LifelongAgentBench, and work measuring test-time learning with human comparison have begun systematically evaluating these capabilities. However, existing models still suffer from limitations in their ability to leverage experience for self-improvement. Our approach addresses this by providing RL training that optimizes memory quality through task performance.

经验驱动与终身学习。 近期工作探索了智能体如何从累积经验中学习。 Reflexion 使用语言自我反思来改进，而 ExpeL 在不同任务间积累洞见。 Dynamic Cheatsheet 通过启发式更新维护演化记忆；ReasoningBank 通过轨迹比较扩展记忆。 SkillWeaver 和 PolySkill 研究面向自我改进或持续学习智能体的可复用技能。 这些工作与我们的设置互为补充：它们主要依赖启发式或基于提示的经验更新，而 MemoPilot 使用下游奖励直接优化记忆更新策略。 EvaLearn、LifelongAgentBench 等基准以及通过人类比较衡量测试时学习的工作，已开始系统评估这些能力。 然而，现有模型利用经验进行自我改进的能力仍存在局限。 我们通过强化学习训练，根据任务表现优化记忆质量，以解决这一问题。

RL for Optimizing Text and Auxiliary Policies. Reinforcement learning has been used to optimize a variety of text artifacts around LLMs. RLPrompt and OPRO optimize prompts for downstream tasks. Prompt-R1 train prompt rewriters via RL. RLAD trains abstraction generators for reasoning, demonstrating that allocating test-time compute to abstraction generation can outperform generating more solutions directly. Xie et al. train critics via RL using a decoupled architecture, while Advisor Models learn lightweight policies to steer black-box LLMs. SPIRAL improves strategic reasoning through self-play RL on the player itself. In contrast, MemoPilot keeps the player frozen and trains an external memory module, making it applicable to stronger or closed-source players without player-side parameter updates. Our work follows this general paradigm of training auxiliary models with RL, but focuses specifically on strategic memory generation with multi-turn training.

用于优化文本和辅助策略的强化学习。 强化学习已被用于优化 LLM 周边的多种文本产物。 RLPrompt 和 OPRO 为下游任务优化提示。 Prompt-R1 通过强化学习训练提示改写器。 RLAD 训练用于推理的抽象生成器，表明把测试时计算分配给抽象生成，可能优于直接生成更多解答。 Xie 等人使用解耦架构通过强化学习训练批评模型，而 Advisor Models 学习轻量策略来引导黑盒 LLM。 SPIRAL 通过直接在玩家上进行自博弈强化学习来改进战略推理。 相比之下，MemoPilot 保持玩家冻结并训练外部记忆模块，因此无需更新玩家侧参数，就能应用于更强或闭源的玩家。 我们的工作遵循使用强化学习训练辅助模型的总体范式，但特别聚焦通过多轮训练生成战略记忆。

7. Limitations

1) Dependence on informative experience and rewards. MemoPilot is designed for settings where past interactions contain reusable signal and downstream reward is available for training. When trajectories are low-information or rewards are extremely sparse, memory updates may have limited evidence to improve from. Auxiliary signals such as token efficiency or trajectory-quality rubrics could provide denser training feedback in such settings.

1）依赖信息充分的经验和奖励。 MemoPilot 面向过去交互包含可复用信号、且训练时可获得下游奖励的场景。 当轨迹信息量较低或奖励极度稀疏时，记忆更新可用于改进的证据可能有限。 token 效率或轨迹质量评分标准等辅助信号，可以在这类场景中提供更密集的训练反馈。

2) Bounded memory capacity. Our experiments use a 512-token memory budget for fair comparison. This budget is a hyperparameter that can be scaled with task requirements, but very long single-task trajectories may still require standard preprocessing such as chunking or summarization before memory updates.

2）记忆容量有限。 为进行公平比较，我们的实验采用 512 token 的记忆预算。 这一预算是可以随任务需求扩展的超参数，但非常长的单任务轨迹在更新记忆前，仍可能需要分块或摘要等标准预处理。

3) Degrade when faced with non-stationary or adaptive opponents. As discussed in Table 7, MemoPilot can degrade when the environment changes faster than its evidence accumulation cycle, especially if a new opponent directly exploits a previously stored belief. This reflects a tradeoff between maintaining stable beliefs under stochasticity and rapidly refining them under distribution shifts.

3）面对非平稳或自适应对手时表现下降。 如表7所述，当环境变化快于证据积累周期时，MemoPilot 的表现可能下降，尤其是新对手直接利用此前存储的信念时。 这反映了在随机性下维护稳定信念与在分布偏移下快速完善信念之间的权衡。

8. Conclusion

We introduce MemoPilot, a framework that treats memory updating as a trainable decision process optimized via multi-turn GRPO. By using turn-level advantage estimation and proxy rewards, our approach stabilizes learning in stochastic environments and significantly outperforms heuristic baselines. On both LHE and RPS, MemoPilot enables frozen LLMs to achieve rapid test-time learning, ranking first in Elo ratings. Furthermore, the learned memory policies demonstrate strong robustness, successfully generalizing to unseen opponents and larger player models without additional parameter updates.

我们提出 MemoPilot，该框架把记忆更新视为可训练的决策过程，并通过多轮 GRPO 进行优化。 通过使用轮次级优势估计和代理奖励，我们的方法稳定了随机环境中的学习，并显著优于启发式基线。 在 LHE 和 RPS 上，MemoPilot 都使冻结 LLM 实现快速测试时学习，并在 Elo 排名中位列第一。 此外，学到的记忆策略表现出很强的稳健性，无需额外参数更新即可成功泛化到未见对手和更大的玩家模型。

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

本文工作的目标是推动机器学习领域的发展。 我们的工作可能产生许多社会影响，但我们认为其中没有需要在此特别强调的事项。