---

ExpeL: LLM Agents Are Experiential Learners

MemoryAgentExperiential Learning230+AAAI 2024CCF-A清华大学北京信息科学与技术国家研究中心

Zhao A, Huang D, Xu Q, et al. ExpeL: LLM Agents Are Experiential Learners. AAAI 2024.

https://andrewzh112.github.io/expel

https://github.com/LeapLabTHU/ExpeL

---

ExpeL：LLM 智能体是经验学习者

Abstract

The recent surge in research interest in applying large language models (LLMs) to decision-making tasks has flourished by leveraging the extensive world knowledge embedded in LLMs. While there is a growing demand to tailor LLMs for custom decision-making tasks, finetuning them for specific tasks is resource-intensive and may diminish the model's generalization capabilities. Moreover, state-of-the-art language models like GPT-4 and Claude are primarily accessible through API calls, with their parametric weights remaining proprietary and unavailable to the public. This scenario emphasizes the growing need for new methodologies that allow learning from agent experiences without requiring parametric updates. To address these problems, we introduce the Experiential Learning (ExpeL) agent.

近期，将大语言模型（LLM）应用于决策任务的研究兴趣激增，这得益于 LLM 所蕴含的广泛世界知识。 尽管定制 LLM 以适应特定决策任务的需求日益增长，但针对具体任务微调模型需要大量资源，并可能削弱模型的泛化能力。 此外，GPT-4 和 Claude 等最先进的语言模型主要通过 API 调用访问，其参数权重属于专有信息，公众无法获取。 这一情形凸显了对新方法日益增长的需求：无需更新参数，也能从智能体经验中学习。 为解决这些问题，我们提出经验学习（Experiential Learning，ExpeL）智能体。

Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions. Our empirical results highlight the robust learning efficacy of the ExpeL agent, indicating a consistent enhancement in its performance as it accumulates experiences. We further explore the emerging capabilities and transfer learning potential of the ExpeL agent through qualitative observations and additional experiments.

我们的智能体从一组训练任务中自主收集经验，并用自然语言抽取知识。 在推理时，智能体会回忆抽取出的洞见和过去经验，以做出有依据的决策。 我们的实证结果凸显了 ExpeL 智能体强健的学习效果，表明随着经验积累，其性能持续提升。 我们还通过定性观察和额外实验，进一步探索 ExpeL 智能体涌现出的能力及其迁移学习潜力。

A computer program is said to learn from experience $E$ with respect to some class of tasks $T$ and performance measure $P$, if its performance at tasks in $T$, as measured by $P$, improves with experience $E$. Tom Mitchell

如果一个计算机程序针对某类任务 $T$ 和性能度量 $P$，其在 $T$ 中任务上的性能按照 $P$ 衡量后，能够随着经验 $E$ 的积累而提高，那么就称该程序从经验 $E$ 中进行了学习。 Tom Mitchell

图1：ExpeL 智能体概览。左：ExpeL 分三个阶段运行：（1）将成功与失败经验收集到经验池中；（2）从这些经验中抽取并抽象跨任务知识；（3）在评估任务中应用获得的洞见，并回忆过去的成功经验。右：（A）通过 Reflexion 收集经验，使智能体能在失败后自我反思并重试任务；（B）洞见抽取步骤。面对成功/失败经验对或长度为 L 的成功经验列表时，智能体使用 ADD、UPVOTE、DOWNVOTE 和 EDIT 操作动态修改已有洞见列表，重点抽取常见失败模式和最佳实践。

1. Introduction

Machine learning research has long been captivated by the potential of autonomous agents and their capabilities. In recent times, incorporating large language models into these agents has unveiled a broad spectrum of applications, even extending beyond academia. One of the significant advantages of LLMs lies in their world knowledge, allowing them to be inherently versatile across various scenarios.

长期以来，自主智能体及其能力所蕴含的潜力一直吸引着机器学习研究。 近年来，将大语言模型融入这些智能体，催生了广泛的应用，甚至延伸到学术界之外。 LLM 的一项显著优势在于其世界知识，使它们天然能够适应各种场景。

On the one hand, previous works investigated finetuning LLMs with a large number of environment interactions or with a large amount of human-labeled datasets. This class of methods incurs high computational costs and needs access to the LLM's parametric weights. Furthermore, finetuning an LLM restricts its functionalities and can hurt its generalization abilities.

一方面，先前工作研究了利用大量环境交互或大量人工标注数据集微调 LLM。 这类方法会产生高昂的计算成本，并且需要访问 LLM 的参数权重。 此外，微调 LLM 会限制其功能，并可能损害其泛化能力。

On the other hand, prompting methods can augment an LLM with better sequential decision-making planning abilities with only a few in-context examples. However, since current LLMs are bounded by context window size, these agents have no recollections of what they have seen, and therefore no learning can be done outside of a few demonstrations. So, how can we strike a balance between these paradigms?

另一方面，提示方法只需少量上下文示例，就能增强 LLM 的序列决策规划能力。 然而，由于当前 LLM 受上下文窗口大小限制，这些智能体无法回忆自己看过的内容，因此除了少数演示之外无法进行学习。 那么，我们如何在这些范式之间取得平衡？

We present the Experiential Learning (ExpeL) agent as a solution. Our agent autonomously gathers experiences from a collection of training tasks through trial and error. From these experiences, it derives natural language insights and employs its own successful experiences as in-context examples during test time. Our agent's learning process is analogous to a student studying for an exam and then taking it on a single attempt, reflecting many real-world situations.

我们提出经验学习（ExpeL）智能体作为解决方案。 我们的智能体通过试错，从一组训练任务中自主收集经验。 它从这些经验中获得自然语言洞见，并在测试时将自身的成功经验用作上下文示例。 我们智能体的学习过程类似于学生为考试复习，然后只参加一次考试，这反映了许多现实情形。

Unlike self-improvement methods like Reflexion, our approach emphasizes the importance of retaining experiences across multiple tasks to enhance agent performance. Moreover, ExpeL learns without parameter updates, making it compatible with powerful closed-source models like GPT-4 or Claude. Lastly, the experience-gathering step does not require a large amount of data or human labels.

与 Reflexion 等自我改进方法不同，我们的方法强调保留跨多个任务的经验对于提升智能体性能的重要性。 此外，ExpeL 无需更新参数即可学习，因此能与 GPT-4 或 Claude 等强大的闭源模型兼容。 最后，经验收集步骤不需要大量数据或人工标签。

We evaluated ExpeL on three vastly different domains and consistently outperformed strong baselines. Additionally, we showcased a transfer learning scenario where our agent that accumulated knowledge from source tasks showed positive forward transfer to target tasks. Finally, we highlighted some unexpected emerged abilities the ExpeL agent gained.

我们在三个截然不同的领域评估了 ExpeL，它始终优于强基线。 此外，我们展示了一个迁移学习场景：从源任务积累知识的智能体，对目标任务表现出正向迁移。 最后，我们重点展示了 ExpeL 智能体获得的一些意外涌现能力。

In summary, our key contributions are as follows: (1) we introduced ExpeL, a novel LLM agent that autonomously learns from experience without gradient updates; (2) We evaluated ExpeL on a diverse set of tasks to showcase its learning abilities and improvement on top of existing planning methods; (3) we showed a novel setting of transfer learning for our LLM agent and demonstrated forward transferability from source tasks to target tasks. Lastly, we believe that as planning algorithms and foundational models continue to improve, ExpeL's paradigm stands to gain significant benefits from their enhanced performances.

总之，我们的主要贡献如下：（1）提出 ExpeL，这是一种无需梯度更新、能够自主从经验中学习的新型 LLM 智能体；（2）在多样化任务上评估 ExpeL，以展示其学习能力以及在现有规划方法基础上的改进；（3）为 LLM 智能体展示了一种新的迁移学习设定，并证明了从源任务到目标任务的正向迁移能力。 最后，我们相信，随着规划算法和基础模型不断改进，ExpeL 范式将从它们提升的性能中显著受益。

2. Related Work

We discuss the most relevant related works in this section. See Appendix for detailed discussions on related works.

本节讨论最相关的工作。 相关工作的详细讨论见附录。

Prompt-based Learning: Prompt-based learning refines label prediction tasks by modifying the input context, facilitating swift adaptation to new tasks with minimal data. This approach capitalizes on LLMs for answers without parameter tuning as they can be augmented using in-context learning. LAMA and GPT-3 are early works that promoted this formulation. Efforts to reduce the intricacies of prompt design include automatic reasoning chains for NLP. Similarly, the ExpeL agent also autonomously learns from experiences using extracted insights and self-generated in-context trajectories by altering the execution prompt.

基于提示的学习： 基于提示的学习通过修改输入上下文来改进标签预测任务，使模型能够利用少量数据快速适应新任务。 这种方法利用 LLM 直接生成答案，无需调整参数，因为可以通过上下文学习对模型进行增强。 LAMA 和 GPT-3 是推动这种形式的早期工作。 降低提示设计复杂度的尝试包括为自然语言处理自动生成推理链。 类似地，ExpeL 智能体也通过修改执行提示，利用抽取出的洞见和自行生成的上下文轨迹，自主地从经验中学习。

Retrieval Augmented Generation (RAG): Retrieval allows LLMs to access databases, mitigating hallucinations. Retrieval has also been used to enhance the capabilities of decision-making agents. In contrast to these works, we focus on retrieving the ExpeL agent's self-generated experiences, thus reducing the dependency on gold examples and leveraging domain-specific corpus.

检索增强生成（RAG）： 检索允许 LLM 访问数据库，从而缓解幻觉。 检索也被用于增强决策智能体的能力。 与这些工作不同，我们专注于检索 ExpeL 智能体自行生成的经验，从而减少对标准示例的依赖，并利用领域特定语料库。

Planning for LLM Agents: Application of LLM agents in fields like robotics, natural sciences, game-playing, and workflows has surged, with emphasis on their world knowledge in fewshot settings. Moreover, LLMs have demonstrated promising zero/few-shot planning and reasoning capabilities in various configurations, including embodied environments and reasoning tasks.

LLM 智能体规划： LLM 智能体在机器人、自然科学、游戏和工作流等领域的应用激增，研究重点在于它们在少样本设定下具备的世界知识。 此外，LLM 已在各种配置中展现出很有前景的零样本或少样本规划与推理能力，包括具身环境和推理任务。

Self-improvement and Memory for LLM Agents: Agents like Reflexion showcase feedback-based improvement, yet often lack cross-task memory. Other agents exhibit potential in persistent memory within multi-agent contexts. Our ExpeL agent combines these approaches, focusing on task-solving while benefiting from self-generated in-context examples and abstracted insights from memory.

LLM 智能体的自我改进与记忆： Reflexion 等智能体展示了基于反馈的改进，但往往缺少跨任务记忆。 其他智能体则展现出在多智能体环境中使用持久记忆的潜力。 我们的 ExpeL 智能体结合了这些方法，在专注于解决任务的同时，受益于自行生成的上下文示例和从记忆中抽象出的洞见。

3. Preliminaries

Complex Interactive Tasks. We work with complex interactive tasks where at each time step $i\in \{0,\dots ,H\}$, the agent receives an observation $o\in \mathcal{O}$, and from its observation history $H_t$ decides to perform action $a\in \mathcal{A}$. The objective of the agent is to achieve some goal $g\in \mathcal{G}$. We only deal with deterministic environments in this work.

复杂交互任务。 我们研究复杂交互任务：在每个时间步 $i\in \{0,\dots ,H\}$，智能体接收观察 $o\in \mathcal{O}$，并根据其观察历史 $H_t$ 决定执行动作 $a\in \mathcal{A}$。 智能体的目标是实现某个目标 $g\in \mathcal{G}$。 本文只处理确定性环境。

Large Language Models. A large language model is a statistical model of the natural language, typically a neural network. In our setting, we use an autoregressive language model, which given an ordered list of existing tokens $\mathbf{x}=\{x_1,x_2,...,x_{l-1}\}$, outputs the probability of the next token $p(x_l\mid x_{<l})$. An instruction-following LLM is typically finetuned on various NLP tasks that are formatted into instruction, input, response tuples. Instruction-tuned models are better at following natural language instructions which alleviates the need for heavy prompt engineering.

大语言模型。 大语言模型是自然语言的统计模型，通常为神经网络。 在我们的设定中，我们使用自回归语言模型：给定已有 token 的有序列表 $\mathbf{x}=\{x_1,x_2,...,x_{l-1}\}$，模型输出下一个 token 的概率 $p(x_l\mid x_{<l})$。 遵循指令的 LLM 通常在各种自然语言处理任务上进行微调，这些任务被格式化为指令、输入和响应三元组。 经过指令微调的模型更善于遵循自然语言指令，从而减轻了繁重提示工程的需要。

ReAct and Reflexion. ReAct and Reflexion are promising frameworks enabling the aforementioned proficiency of LLMs in reasoning and self-improvement. ReAct explicitly intertwines observations, actions, and thoughts, providing a foundation for robust planning and reasoning capabilities. Building upon it, Reflexion introduces an additional reflective step before reattempting the subsequent trial of the same task, enhancing the model's adaptive learning process.

ReAct 与 Reflexion。 ReAct 和 Reflexion 是很有前景的框架，使 LLM 能够具备上述推理和自我改进能力。 ReAct 明确地将观察、动作和思考交织起来，为强健的规划与推理能力奠定基础。 Reflexion 在此基础上，于再次尝试同一任务之前引入额外的反思步骤，从而增强模型的自适应学习过程。

4. ExpeL: An Experiential Learning Agent

Recent advancements in generative LLMs suggest an intriguing approach. Rather than altering the LLM parameters, adjusting the prompts may be more beneficial: this strategy ensures that the LLM's inherent common sense knowledge remains intact, allowing for superior generalization. Furthermore, some of the most potent language models are proprietary. Thus, focusing on prompt-based methods seems promising as a way to harness the strengths of these advanced LLMs.

生成式 LLM 的最新进展提示了一种引人关注的方法。 与其改变 LLM 参数，调整提示可能更有益：这种策略能确保 LLM 固有的常识知识保持完整，从而实现更好的泛化。 此外，一些最强大的语言模型是专有模型。 因此，专注于基于提示的方法，似乎是利用这些先进 LLM 优势的一条很有前景的路径。

Additionally, previous works on learning in LLM agents have primarily been trained on extensive human-labeled datasets or improved via iterative retries on a single task. A relatively less explored area is facilitating agents to learn autonomously from their own experiences, similar to a student gaining insights from practicing for an exam. The student tackles practice problems multiple times to derive insights. At the exam, the student rely solely on these insights and draw memories of similar problems to answer the questions with one attempt. With this in mind, we wish to design an LLM agent that autonomously gathers experiences and extracts insights, then uses these cross-task insights and memories of similar tasks to aid its decision-making.

此外，先前关于 LLM 智能体学习的工作，主要使用大规模人工标注数据集进行训练，或通过在单个任务上迭代重试来改进。 一个相对较少被探索的方向，是让智能体能够自主地从自身经验中学习，就像学生通过考前练习获得洞见一样。 学生会多次处理练习题，从中获得洞见。 在考试时，学生仅依靠这些洞见，并调用对类似问题的记忆，尝试一次就回答问题。 基于这一想法，我们希望设计一种能够自主收集经验并抽取洞见的 LLM 智能体，然后利用这些跨任务洞见和对类似任务的记忆来辅助决策。

We aim to enhance a planning LLM agent, such as ReAct, with learning abilities that allow it to improve through inter-task experiences without any parameter updates. Inspired by the cognitive abilities inherent in human learning, as well as the benefits observed in self-learning autonomous agents and the progress made in prompt-based methods, we developed the Experiential Learning (ExpeL) agent. During the training stage, the agent interacts with the environment, gathering experiences via trial and error. These experiences are stored in an experience pool. From this pool, the agent later extracts insights, similar to off-policy learning, in which the agent can learn from experiences of a behavior policy.

我们的目标是为 ReAct 等规划型 LLM 智能体增加学习能力，使其无需任何参数更新，就能通过跨任务经验得到改进。 受到人类学习固有认知能力、自学习自主智能体所展现益处以及基于提示方法进展的启发，我们开发了经验学习（ExpeL）智能体。 在训练阶段，智能体与环境交互，通过试错收集经验。 这些经验被存入经验池。 随后，智能体从该经验池中抽取洞见，这类似于离策略学习，即智能体可以从行为策略产生的经验中学习。

During the evaluation stage, the agent attempts unseen tasks with a single try, augmented with extracted insights and successful trajectories in its experience pool gathered from the training stage. Refer to Figure 1 for detailed information on our agent framework.

在评估阶段，智能体只尝试一次未见任务，同时利用抽取出的洞见和训练阶段收集到经验池中的成功轨迹进行增强。 我们的智能体框架详情见 图1。

4.1 Gathering Experiences

To gather diverse experiences that can be useful to extract information from, we leverage Reflexion to continuously retry the training task at most $Z$ times. In particular, the agent will be given a training task $t_n$ at the $z$-th trial, fewshot examples $F_{\text{manual}}$ and past reflections ${\nu }_{n,z}$ (initially, ${\nu }_{n,0}$ is the empty string). At first, the agent will attempt the task with fewshot examples concatenated with its current trajectory ${\tau }_{n,0}$ as the context, and use ReAct as the base planning algorithm, ${\text{LLM}}_{\text{ReAct}}(\cdot \mid {\tau }_{n,0},F_{\text{manual}},{\nu }_{n,0})$.

为了收集有助于抽取信息的多样化经验，我们利用 Reflexion 连续重试训练任务，最多重试 $Z$ 次。 具体而言，在第 $z$ 次尝试时，智能体会获得训练任务 $t_n$、少样本示例 $F_{\text{manual}}$ 和过去的反思 ${\nu }_{n,z}$（初始时，${\nu }_{n,0}$ 为空字符串）。 开始时，智能体会把少样本示例与当前轨迹 ${\tau }_{n,0}$ 拼接起来作为上下文，并使用 ReAct 作为基础规划算法，即 ${\text{LLM}}_{\text{ReAct}}(\cdot \mid {\tau }_{n,0},F_{\text{manual}},{\nu }_{n,0})$。

On the $z$-th trial, when the agent finishes the task or the maximum number of steps $H$ is reached, the ExpeL agent's experience pool $\mathcal{B}$ ingests the trajectory ${\tau }_{n,z}$. Then, if the agent succeeds, it moves on to the next task. However, if the agent fails, it will look at its failed trajectory and self-reflect to produce ${\nu }_{n,z+1}=\mathrm{concat}({\nu }_{n,z},{\text{LLM}}_{\text{reflect}}({\tau }_{n,z}))$ to see where it can do better on the next retry, concatenated with the previous reflections. In the next retry, the agent will augment its context with reflection ${\nu }_{n,z+1}$, the input to the LLM policy, ${\text{LLM}}_{\text{ReAct}}(\cdot \mid {\tau }_{n,z+1},F_{\text{manual}},{\nu }_{n,z+1})$.

在第 $z$ 次尝试中，当智能体完成任务或达到最大步数 $H$ 时，ExpeL 智能体的经验池 $\mathcal{B}$ 会接收轨迹 ${\tau }_{n,z}$。 如果智能体成功，它便继续下一个任务。 但如果智能体失败，它会查看失败轨迹并进行自我反思，生成 ${\nu }_{n,z+1}=\mathrm{concat}({\nu }_{n,z},{\text{LLM}}_{\text{reflect}}({\tau }_{n,z}))$，以分析下次重试时可以改进之处，并将其与先前反思拼接起来。 在下一次重试中，智能体会用反思 ${\nu }_{n,z+1}$ 增强上下文，作为 LLM 策略的输入，即 ${\text{LLM}}_{\text{ReAct}}(\cdot \mid {\tau }_{n,z+1},F_{\text{manual}},{\nu }_{n,z+1})$。

To highlight, this trial and error way of gathering experiences not only improves the chances of getting more positive examples for experience recall during evaluation but also allows for collecting valuable success/failure pairs used for comparisons during insight extraction (Section 4.2). The pseudo-code can be found in Algorithm 1.

需要强调的是，这种通过试错收集经验的方式，不仅提高了获得更多正例以供评估时回忆经验的机会，还能收集有价值的成功/失败经验对，用于洞见抽取过程中的比较（第 4.2 节）。 伪代码见 算法1。

Algorithm 1: ExpeL - Experience Gathering

Initialize: policy ${\text{LLM}}_{\text{ReAct}}$; self-reflection model ${\text{LLM}}_{\text{reflect}}$; training tasks ${\mathcal{T}}_{\text{train}}$; fewshot examples $F_{\text{manual}}$; experience pool $\mathcal{B}\leftarrow F_{\text{manual}}$; task count $N$; retry limit $Z$; step limit $H$; task index $n\leftarrow 1$

1. while task $n\le N$ do
2. $t_n\leftarrow {\mathcal{T}}_{\text{train}}[n]$
3. ${\nu }_{n,0}\leftarrow \text{“"}$
4. $$ for trial $z=0$ to $Z$ do
5. $o_0\leftarrow \mathtt{\text{env.reset}}(t_n)$; ${\tau }_{n,z}\leftarrow o_0$
6. $$ for timestep $i=0$ to $H$ do
7. $\begin{array}{rl}{a}_i\leftarrow {\text{LLM}}_{\text{ReAct}}(& a_i\mid {\tau }_{n,z},\\ & F_{\text{manual}},{\nu }_{n,z})\end{array}$
8. $o_{i+1},r_{i+1},\mathtt{\text{done}}\leftarrow \mathtt{\text{env.step}}(a_i)$
9. ${\tau }_{n,z}\leftarrow {\tau }_{n,z}\cup \{(o_i,a_i,o_{i+1},r_{i+1})\}$
10. $$ if $\mathtt{\text{done}}$ then break
11. $$ end for
12. $\mathcal{B}\leftarrow \mathcal{B}\cup {\tau }_{n,z}$
13. $$ if $\mathtt{\text{done}}$ or $z=Z$ then $n\leftarrow n+1$; break
14. $$ else $\begin{array}{rl}{\nu }_{n,z+1}\leftarrow \mathrm{concat}(& {\nu }_{n,z}\\ & +{\text{LLM}}_{\text{reflect}}({\tau }_{n,z}))\end{array}$
15. $$ end for
16. end while
17. return $\mathcal{B}$

4.2 Learning from Experiences

Human learning occurs mainly either by storing successful trajectories in memory, which can be later recalled as specific examples, or by extracting high-level insights from experiences, enabling generalization to novel situations. ExpeL considers both of these learning modes to boost task performance. Concretely, an instruction $I$ given to an LLM agent can be broken down into task specifications and fewshot examples. We can augment task specifications with an agent's extracted insights from past experiences, where an instruction-following LLM can be leveraged to follow them closely.

人类学习主要通过两种方式发生：将成功轨迹存储在记忆中，以便之后将其作为具体示例回忆；或者从经验中抽取高层洞见，从而泛化到新情形。 ExpeL 同时考虑这两种学习模式，以提升任务性能。 具体而言，给定 LLM 智能体的一条指令 $I$ 可以分解为任务规范和少样本示例。 我们可以利用智能体从过去经验中抽取的洞见来增强任务规范，并借助遵循指令的 LLM 严格遵循这些洞见。

For fewshot examples, we can allow the agent to retrieve from its experience pool with top-$k$ relevant examples to aid its decisions. Next, we detail our experience recall and insight extraction mechanisms.

对于少样本示例，我们可以让智能体从经验池中检索 top-$k$ 个相关示例，以辅助其决策。 下面，我们将详细介绍经验回忆和洞见抽取机制。

Similar Experiences as Demonstrations. Works have shown that using in-context examples that are semantically similar to the task at hand results in better performance. Moreover, when involved in a novel situation, humans also recall from their memory similar tasks they've solved as references when attempting the task. Motivated by these observations, we propose experience recall to retrieve successful trajectories from the experience pool gathered during training based on task similarity.

将相似经验用作演示。 已有工作表明，使用与当前任务语义相似的上下文示例能带来更好的性能。 此外，在面对新情形时，人类也会从记忆中回想已经解决过的类似任务，作为尝试当前任务时的参考。 受这些观察启发，我们提出经验回忆：依据任务相似性，从训练期间收集的经验池中检索成功轨迹。

Concretely, we used the Faiss vectorstore as the experience pool, kNN retriever and all-mpnet-base-v2 embedder to obtain top-$k$ successful trajectories that have the maximum inner-product task similarity with the evaluation task. The advantage of using task similarity as the retrieval rank is that if the agent repeats a task or does a task similar to an existing successful trajectory from the experience pool, the agent only needs to closely imitate the successful trajectory and have less burden on ability extrapolation.

具体而言，我们使用 Faiss 向量存储作为经验池，以 kNN 作为检索器，并使用 all-mpnet-base-v2 嵌入模型，获取与评估任务具有最大内积任务相似度的 top-$k$ 条成功轨迹。 使用任务相似性作为检索排序依据的优势是：如果智能体重复某项任务，或执行与经验池中已有成功轨迹相似的任务，它只需紧密模仿该成功轨迹，从而减轻能力外推的负担。

Learning from Successes and Failures. To leverage the diverse outcomes gathered during the experience collection phase, we believe the agent should analyze experiences in two distinct ways. First, we let the agent compare a failed trajectory with a successful trajectory for the same task. This comparison offers a concrete understanding of the agent's shortcomings, highlighting the correct and incorrect actions. Second, we let the agent identify patterns within a set of successful trajectories from different tasks. This approach sheds light on common "good practices" that the agent can adopt to ensure success in evaluation tasks.

从成功与失败中学习。 为了利用经验收集阶段获得的多样化结果，我们认为智能体应该以两种不同方式分析经验。 首先，我们让智能体比较同一任务的一条失败轨迹与一条成功轨迹。 这种比较使智能体能够具体理解自身不足，并突出正确和错误的动作。 其次，我们让智能体从不同任务的一组成功轨迹中识别模式。 这种方法揭示了智能体可采用的通用“良好实践”，以确保在评估任务中取得成功。

图2：洞见抽取提示模板。ExpeL 智能体用于洞见抽取的提示模板；同一模板既用于成功/失败经验对（A，黄色），也用于大小为 L 的成功经验集合（B，绿色）。

For the implementation, we give the agent's instruction-following ${\text{LLM}}_{\text{insights}}$ several operators to apply on an existing set of insights $\hat{\iota }$. We initialize the set of insights to an empty set $\hat{\iota }=\mathrm{\varnothing }$ and iteratively provide the LLM with fail/success pairs or lists of $L$ successes (created by sampling without replacement) from the experience pool. The operations the LLM can perform are: ADD a new insight, EDIT the content of an existing insight, DOWNVOTE to disagree with an existing insight, or UPVOTE to agree with an existing insight.

在实现中，我们为智能体中遵循指令的 ${\text{LLM}}_{\text{insights}}$ 提供多个操作符，用于处理已有洞见集合 $\hat{\iota }$。 我们将洞见集合初始化为空集 $\hat{\iota }=\mathrm{\varnothing }$，并迭代地向 LLM 提供来自经验池的失败/成功经验对，或包含 $L$ 条成功经验的列表（通过无放回采样创建）。 LLM 可以执行的操作包括：ADD 一条新洞见、EDIT 已有洞见的内容、用 DOWNVOTE 表示不同意已有洞见，或用 UPVOTE 表示同意已有洞见。

A newly added insight will have an initial importance count of two associated with it, and the count will increment if subsequent operators UPVOTE or EDIT are applied to it and will decrement when DOWNVOTE is applied to it. If an insight's importance count reaches zero, it will be removed. This particular design choice robustifies the process since even successful trajectories can be suboptimal and mislead the generated insights. The prompt template we used can be found in Figure 2.

每条新加入的洞见都关联一个初始值为 2 的重要性计数；若后续对它执行 UPVOTE 或 EDIT，计数就会增加，而执行 DOWNVOTE 时计数会减少。 如果一条洞见的重要性计数降至 0，它将被移除。 这一特定设计使流程更加稳健，因为即便成功轨迹也可能是次优的，并误导所生成的洞见。 我们使用的提示模板见 图2。

We kept the maximum size for a list of successes to $L$ and used gpt-4-0613 as the default ${\text{LLM}}_{\text{insights}}$. We empirically found that gpt-4-0613 is better than gpt-3.5-turbo-0613 at following instructions on how to use the insight extraction operators and hallucinated less. Pseudo-code for this process can be found in Algorithm 2. Finally, ExpeL utilizes these generated insights $\hat{\iota }$ in the task inference phase, described next.

我们将成功经验列表的最大大小设为 $L$，并使用 gpt-4-0613 作为默认的 ${\text{LLM}}_{\text{insights}}$。 我们通过经验观察发现，gpt-4-0613 比 gpt-3.5-turbo-0613 更善于遵循洞见抽取操作符的使用指令，并且产生的幻觉更少。 这一过程的伪代码见 算法2。 最后，ExpeL 在下面介绍的任务推理阶段使用这些生成的洞见 $\hat{\iota }$。

Algorithm 2: ExpeL - Insight Extraction

Initialize: experience pool $\mathcal{B}$ from Algorithm 1; insight extraction model ${\text{LLM}}_{\text{insights}}$; insight set $\hat{\iota }\leftarrow \mathrm{\varnothing }$

1. Divide successes in $\mathcal{B}$ into $L$-sized chunks:
2. $\begin{array}{rl}{C}_{\text{success}}=& \{\{{\tau }_1^{\text{success}},\dots ,{\tau }_L^{\text{success}}\},\\ & \{{\tau }_{L+1}^{\text{success}},\dots ,{\tau }_{2L}^{\text{success}}\},\dots \}\end{array}$
3. Construct fail/success tuples for the same tasks in $\mathcal{B}$:
4. $\begin{array}{rl}{C}_{\text{compare}}=& \{({\tau }_1^{\text{success}},{\tau }_{1,0}^{\text{fail}}),\\ & ({\tau }_1^{\text{success}},{\tau }_{1,1}^{\text{fail}}),\dots ,\\ & ({\tau }_2^{\text{success}},{\tau }_{2,0}^{\text{fail}}),\dots \}\end{array}$
5. for each $c_{\text{compare}}$ in $C_{\text{compare}}$ do
6. $\hat{\iota }\leftarrow {\text{LLM}}_{\text{insights}}(c_{\text{compare}},\hat{\iota })$
7. end for
8. for each $c_{\text{success}}$ in $C_{\text{success}}$ do
9. $\hat{\iota }\leftarrow {\text{LLM}}_{\text{insights}}(c_{\text{success}},\hat{\iota })$
10. end for
11. return $\hat{\iota }$

4.3 Task Inference

After the agent gathers experiences, extracts insights from them, and sets up a vectorstore of successful trajectories, it can proceed to the evaluation. For each task, the task specifications will be augmented with the concatenation of the full list of extracted insights $\hat{\iota }=\mathrm{concat}({\iota }_1,{\iota }_2,{\iota }_3,...)$, and the top-$k$ trajectories with the highest task similarity will be retrieved and used as fewshot in-context examples, $F_{\text{similar tasks}}$. Figure 3 shows an example prompt template structure, and a pseudo-code for this step can be found in Algorithm 3. We believe as the list of extracted insights grows, retrieval could be a feasible solution to manage the context window size.

智能体收集经验、从中抽取洞见并建立成功轨迹向量存储后，就可以进入评估阶段。 对于每项任务，任务规范会用完整抽取洞见列表的拼接结果 $\hat{\iota }=\mathrm{concat}({\iota }_1,{\iota }_2,{\iota }_3,...)$ 进行增强；同时，检索任务相似度最高的 top-$k$ 条轨迹，并将其用作少样本上下文示例 $F_{\text{similar tasks}}$。 图3 展示了一个提示模板结构示例，这一步的伪代码见 算法3。 我们认为，随着抽取洞见列表不断增长，检索可能是管理上下文窗口大小的一种可行方案。

图3：任务推理提示模板。图中展示了 ExpeL 在评估期间的提示模板；白色区域与基础 ReAct 智能体的输入相同，紫色区域则是 ExpeL 的差异所在：加入从过去经验中抽取的洞见，并依据任务相似性动态检索过去的成功上下文示例。

Algorithm 3: ExpeL - Evaluation

Initialize: ExpeL agent ${\text{LLM}}_{\text{ExpeL}}$; text embedder $\mathcal{E}$; experience pool $\mathcal{B}$; insight set $\hat{\iota }$; evaluation tasks ${\mathcal{T}}_{\text{evaluation}}$; task count $M$; fewshot count $k$; successes $S\leftarrow 0$

1. for task $m=1$ to $M$ do
2. $t_m\leftarrow {\mathcal{T}}_{\text{evaluation}}[m]$
3. $o_0\leftarrow \mathtt{\text{env.reset}}(t_m)$; ${\tau }_m\leftarrow o_0$
4. $\begin{array}{rl}{F}_{\text{similar tasks}}\leftarrow \mathtt{\text{Faiss}}(& t_m,\mathcal{B},\\ & \mathcal{E},k)\end{array}$
5. $$ for timestep $i=1$ to $H$ do
6. $\begin{array}{rl}{a}_i\leftarrow {\text{LLM}}_{\text{ExpeL}}(& a_i\mid {\tau }_m,\\ & F_{\text{similar tasks}},\hat{\iota })\end{array}$
7. $o_{i+1},r_{i+1},\mathtt{\text{done}}\leftarrow \mathtt{\text{env.step}}(a_i)$
8. ${\tau }_m\leftarrow {\tau }_m\cup \{(o_i,a_i,o_{i+1},r_{i+1})\}$
9. $$ if $\mathtt{\text{done}}$ then break
10. $$ end for
11. $$ if $r_{i+1}=1$ then $S\leftarrow S+1$
12. end for
13. return $S/M$

4.4 Transfer Learning

After demonstrating how learning by using experiences from a training set can benefit an LLM agent in solving an unseen task in the same task distribution, we investigate another interesting setting where knowledge accumulated from a source task distribution could be useful for a target task distribution with minimal target task examples for the ExpeL agent. Like most transfer learning settings, we assume that the source and target tasks exhibit common knowledge. Therefore, experiences accumulated from source tasks can benefit the agent in solving a new set of target tasks.

在证明利用训练集经验进行学习能够帮助 LLM 智能体解决同一任务分布中的未见任务后，我们进一步研究另一种有趣的设定：在目标任务示例极少的情况下，从源任务分布积累的知识能否帮助 ExpeL 处理目标任务分布。 与大多数迁移学习设定一样，我们假设源任务与目标任务包含共同知识。 因此，从源任务积累的经验可以帮助智能体解决一组新的目标任务。

Similar to pretraining on source task and finetuning on target task in transfer learning literature, we propose to use the extracted insights $\hat{\iota }$ from the source task and fewshot examples from the target task to "finetune" the insights so that they are more applicable in the target task. We hypothesize that using target task fewshot examples can better ground the insights into the target task and mitigate hallucinations. An example prompt template to "finetune" extracted insights from a source domain to tailor them to a target domain is illustrated in Figure 4.

类似于迁移学习文献中在源任务上预训练、在目标任务上微调，我们提出使用从源任务抽取的洞见 $\hat{\iota }$ 和来自目标任务的少样本示例，对洞见进行“微调”，使其更适用于目标任务。 我们假设，使用目标任务的少样本示例可以使洞见更好地扎根于目标任务，并缓解幻觉。 将从源领域抽取的洞见“微调”为适用于目标领域的示例提示模板见 图4。

图4：迁移学习微调提示模板。该提示模板用于将知识从源领域微调到目标领域；灰色高亮区域应填写简洁的任务描述。

图5：主结果。图中给出 HotpotQA、ALFWorld 和 WebShop 三个不同领域的平均任务成功率，灰色箭头表示标准误差。ReAct 和 Act 作为基线；ExpeL 在所有领域都持续优于基线，凸显从经验中学习的重要性。图中还比较了 ExpeL、仅检索 ExpeL 和仅洞见 ExpeL，表明洞见抽取与任务相似性检索都不可或缺，并具有协同作用。

4.5 ExpeL's Strengths

In this section, we outline the key strengths of our framework. First and foremost, ExpeL offers inherent interpretability, as both the extracted experiences and successful trajectories are presented in natural language. This design allows users to easily inspect, modify, or remove potentially harmful trajectories/insights — a challenge in finetuned models. Moreover, users can seamlessly add expert insights or trajectories to an ExpeL agent. Additionally, our learning approach is highly accessible; it demands less data, reduces computational resources, and is straightforward to implement.

本节概述我们框架的主要优势。 首先，ExpeL 具有内在可解释性，因为抽取出的经验和成功轨迹都以自然语言呈现。 这一设计允许用户轻松检查、修改或移除可能有害的轨迹或洞见，而这对于经过微调的模型而言很困难。 此外，用户可以无缝地向 ExpeL 智能体加入专家洞见或轨迹。 我们的学习方法也很容易使用：所需数据更少、计算资源更低，而且实现直接。

Furthermore, self-improvement methods like Reflexion facilitate intra-task improvements, but ExpeL enables inter-task learning. ExpeL does not rely on retries during deployment, which certain domains require. On the flexibility front, the ExpeL agent boasts a significant level of versatility. It is not restricted to specific language models and complements existing strategies aimed at enhancing LLM agent planning capabilities. Moreover, when applied in conjunction with them, ExpeL might even improve the capabilities of finetuned agents.

此外，Reflexion 等自我改进方法促进任务内改进，而 ExpeL 支持跨任务学习。 ExpeL 在部署期间不依赖重试，而某些领域正需要这种特性。 在灵活性方面，ExpeL 智能体具有显著的通用性。 它不受特定语言模型限制，并能补充旨在增强 LLM 智能体规划能力的现有策略。 此外，当与这些策略结合使用时，ExpeL 甚至可能提升经过微调的智能体能力。

Another strength lies in continuous improvement. Our method stands to benefit from the ongoing enhancements in foundational models. As an illustration, our experiments show that using gpt-4 to extract insights outperforms gpt-3.5-turbo (refer to Section 5.6). Lastly, we introduced a method for transferring extracted insights across domains using only a small amount of finetuning examples, demonstrating the advantage of our approach in diverse settings with limited data.

另一项优势在于持续改进。 我们的方法能够受益于基础模型不断提升的能力。 例如，我们的实验表明，使用 gpt-4 抽取洞见的效果优于 gpt-3.5-turbo（见第 5.6 节）。 最后，我们提出了一种只使用少量微调示例，就能在不同领域之间迁移抽取洞见的方法，展示了我们的方法在数据有限的多样化场景中的优势。

5. Experiments

5.1 Experimental Setup

In line with ReAct, the experiments are designed based on four text-based benchmarks: HotpotQA, a knowledge-intensive dataset that challenges an agent to perform reasoning and question answering using the search tool Wikipedia Docstore API, ALFWorld and WebShop that require the agent to perform interactive multi-step decision-making tasks in respectively a household and an online shopping website environments, and FEVER, that focuses on fact verification tasks using the same API as HotpotQA which makes it suitable for knowledge transfer (Section 5.4). All experiments use four-fold validation, and we report the mean and standard error over the folds.

与 ReAct 一致，实验基于四个文本基准设计：HotpotQA 是一个知识密集型数据集，要求智能体使用 Wikipedia Docstore API 搜索工具进行推理和问答；ALFWorld 和 WebShop 分别要求智能体在家庭环境和在线购物网站环境中执行交互式多步决策任务；FEVER 专注于事实验证任务，使用与 HotpotQA 相同的 API，因此适合知识迁移（第 5.4 节）。 所有实验都使用四折验证，我们报告各折的均值和标准误差。

Following ReAct, for all environments, we use success rate as the evaluation metric: exact matching for HotpotQA and FEVER, completing the task in time for ALFWorld, and purchasing the item that matches all attributes for WebShop. Some additional metrics are introduced when the environment offers them: mean reward (calculated using Equation 1 in Appendix) score $r\in [0,1]$ for WebShop and a score breakdown per task type for ALFWorld.

遵循 ReAct，我们在所有环境中都使用成功率作为评估指标：HotpotQA 和 FEVER 使用精确匹配，ALFWorld 要求在时限内完成任务，WebShop 则要求购买符合全部属性的商品。 当环境提供额外指标时，我们也会使用这些指标：WebShop 使用平均奖励分数 $r\in [0,1]$，ALFWorld 则按任务类型报告分数明细。

We use ReAct and Act as main baselines planning LLM agents, where Act does not have the reasoning steps like ReAct. All agents, including ExpeL, used gpt-3.5-turbo-0613 when performing actions during evaluation. All text generations were done with temperature 0 and greedy decoding. Imitation learning (IL) results were taken from the ReAct paper. More details about the experimental setup can be found in Appendix.

我们使用 ReAct 和 Act 作为主要的规划型 LLM 智能体基线，其中 Act 不包含 ReAct 那样的推理步骤。 包括 ExpeL 在内的所有智能体在评估期间执行动作时，都使用 gpt-3.5-turbo-0613。 所有文本生成均采用温度 0 和贪心解码。 模仿学习（IL）结果取自 ReAct 论文。 实验设置的更多细节见附录。

5.2 Main Results

The primary findings of this study are presented in Figure 5. IL-based method struggles to efficiently perform in WebShop and ALFWorld, possibly due to their demand for more substantial prior and reasoning abilities, which conventional trainings from scratch fail to provide. This limitation shows the promise of leveraging knowledge-based language models to address these challenges. The following claims were made based on (1) a deep understanding of each environment; (2) extracted insights and retrievable in-context examples; and (3) statistics (e.g. number of invalid actions per trial) of the runs.

本研究的主要发现见 图5。 基于模仿学习的方法很难在 WebShop 和 ALFWorld 中高效执行，这可能是因为这些任务需要更丰富的先验知识和推理能力，而传统的从零训练无法提供这些能力。 这一局限表明，利用具备知识的语言模型来应对这些挑战很有前景。 以下结论基于：（1）对每个环境的深入理解；（2）抽取出的洞见和可检索的上下文示例；（3）运行统计信息，例如每次尝试中的无效动作数量。

Experiential learning. Augmenting agents with abstracted insights and the ability to recall successful trajectories improve performance across all environments compared to baseline agents. When restricting the ExpeL agent to only one mode of learning (insights-only or retrieval-only), HotpotQA and ALFWorld environments demonstrate contrasting quantitative distinctions (36%/31% and 50%/55% for HotpotQA and ALFWorld, respectively). The prominent influence of insights on HotpotQA can be due to its reliance on analysing (Wikipedia results) abilities. This highlights the need for general guidelines across various question types.

经验学习。 与基线智能体相比，利用抽象洞见和成功轨迹回忆能力增强智能体，可以提高其在所有环境中的性能。 当 ExpeL 智能体被限制为只使用一种学习模式（仅洞见或仅检索）时，HotpotQA 和 ALFWorld 呈现出相反的量化差异：二者分别为 36%/31% 和 50%/55%。 洞见对 HotpotQA 的显著影响，可能源于该任务对分析 Wikipedia 结果能力的依赖。 这凸显了针对各种问题类型制定通用指导原则的必要性。

Conversely, ALFWorld's task completion, dependent on specific action sets, is better derived from past experiential trajectories. Furthermore, WebShop presents a unique challenge, requiring both website-based reasoning (price comparisons, query reformulation, etc.) and precise execution of actions (searching, clicking, option selection, etc.). Consequently, the performance across these tasks shows a near equilibrium, as reflected in both the success rate and score (37%/38% and 0.675/0.67 for insights/retrieve-only respectively, see Table in Appendix for scores). These observations highlight the synergistic interplay between abstraction and recollection in experiential learning, with ExpeL showing a quantitative advantage over baseline/restricted learning mode agents.

相反，ALFWorld 的任务完成依赖特定动作集合，因此更适合从过去的经验轨迹中学习。 此外，WebShop 带来了一项独特挑战：既需要基于网站的推理，例如价格比较和查询重写，也需要准确执行搜索、点击和选项选择等动作。 因此，这两种学习模式在该任务上的性能近乎均衡，成功率和分数分别为 37%/38% 和 0.675/0.67；分数见附录表格。 这些观察凸显了经验学习中抽象与回忆之间的协同作用，而 ExpeL 相对于基线和受限学习模式智能体表现出量化优势。

Cross-task learning. Another important finding we observe is the comparison with the Reflexion agent. ExpeL matches Reflexion's performance (40% at R3 vs. 39%) for HotpotQA and even outperforms it for ALFWorld (54% at R3 vs. 59%) without repeated attempts. While Reflexion improves results by iteratively refining insights through repeated task execution (R1, R2, R3...), our ExpeL agent leverages cross-task learning by accumulating task experience. However, it is noteworthy that there remains room for improvement in the context of WebShop tasks, approaching the lower side of Reflexion's success rates.

跨任务学习。 我们观察到的另一项重要发现来自与 Reflexion 智能体的比较。 在不重复尝试的情况下，ExpeL 在 HotpotQA 上达到与 Reflexion 相当的性能（R3 时 40%，ExpeL 为 39%），在 ALFWorld 上甚至优于它（R3 时 54%，ExpeL 为 59%）。 Reflexion 通过反复执行任务来迭代改进洞见（R1、R2、R3……），而我们的 ExpeL 智能体则通过积累任务经验来利用跨任务学习。 不过值得注意的是，在 WebShop 任务中仍有改进空间，其性能接近 Reflexion 成功率的较低一侧。

5.3 Agent Behavioral Analysis

In this section, we highlight some observations made by manually inspecting the trajectories of ReAct agents and ExpeL agents, and by pinpointing possible causes of how some unexpected behaviors might have emerged. Please visit the paper's webpage, https://andrewzh112.github.io/expel, for full trajectory demos illustrating the following findings.

本节重点介绍我们通过人工检查 ReAct 智能体和 ExpeL 智能体的轨迹所获得的一些观察，并指出某些意外行为可能如何涌现的原因。 展示以下发现的完整轨迹演示见论文项目页面：https://andrewzh112.github.io/expel。

Hypothesis Formulation & Constraints Adaptation. After extracting the insights from experiences gathered in the training set, we noticed the agent subsequently gained the ability to reassess its whole trajectory in the last steps and conclusively end the task rather than expressing its ineptitude in providing a solution. This ability was particularly observed in HotpotQA (Figures in Appendix) where a likely influential insight was stating that the agent should "consider the answer might be in the observations already made". Therefore the agent would finish by proposing the most probable answer given its past observations rather than concluding with "Unknown" or "Information not available".

假设形成与约束适应。 从训练集收集的经验中抽取洞见后，我们注意到，智能体随后获得了在最后几步中重新评估整条轨迹并明确结束任务的能力，而不是表示自己无法给出解决方案。 这一能力在 HotpotQA 中尤为明显（见附录中的图）；一个可能影响较大的洞见指出，智能体应当“考虑答案可能已经存在于此前的观察中”。 因此，智能体会根据过去观察提出最可能的答案，而不是以“Unknown”或“Information not available”结束。

World Model Belief Update. We noticed our ExpeL agent updated its beliefs through the insights and over its gained experience. This belief thereby update enables the agent to avoid unnecessary actions and increase efficiency in solving a given task. For example, in ALFWorld, the agent completely changed the priors it had in ReAct on the likely locations of a pan (from drawers/countertops/cabinets to stoveburners). This behavior emerged from the extracted insight claiming that "when searching for an item" it needs to "consider its nature and its typical usage" (Figure in Appendix), leading the agent to promptly and accurately find the correct item at the first step while the ReAct agent could not find it in time.

世界模型信念更新。 我们注意到，ExpeL 智能体会通过洞见并随着经验积累更新其信念。 这种信念更新使智能体能够避免不必要的动作，并提高解决给定任务的效率。 例如，在 ALFWorld 中，智能体彻底改变了 ReAct 中关于平底锅可能位置的先验：从抽屉、台面和橱柜，转向炉灶。 这一行为源于抽取出的洞见：在“寻找物品时”，需要“考虑其性质和典型用途”（见附录中的图）；因此，智能体能够在第一步就迅速准确地找到正确物品，而 ReAct 智能体未能及时找到它。

Self-correction. Although ReAct was sometimes not able to reassess its situation when attempting to solve a task, ExpeL demonstrated its proficiency in identifying and rectifying missteps. Notably, when incorrectly taking an object in ALFWorld, the agent has shown its ability to put it back and resume the task by searching for the proper object (Figure in Appendix). This highlights ExpeL's capacity to recover from errors and stay on course without hallucinating when completing tasks. This behavior is possibly encouraged by the generated insight "reassess the situation and consider alternative actions" if "an attempt does not progress the task".

自我纠正。 尽管 ReAct 在尝试解决任务时有时无法重新评估自身处境，ExpeL 却展现了识别并纠正失误的能力。 尤其是在 ALFWorld 中错误拿取一个物品时，智能体表现出将其放回并通过寻找正确物品恢复任务的能力（见附录中的图）。 这凸显了 ExpeL 从错误中恢复、保持任务方向并避免在完成任务时产生幻觉的能力。 当“一次尝试没有推动任务进展”时，生成的洞见“重新评估情况并考虑替代动作”可能促进了这种行为。

5.4 Transfer Learning

In this experiment, we use the HotpotQA dataset as source tasks and the FEVER dataset as target tasks. Like the HotpotQA dataset, we equip the agent with the ability to navigate on Wikipedia using a Docstore API; therefore, we hypothesize that some of the knowledge obtained from HotpotQA tasks should also be beneficial when transferred to the FEVER tasks. We use gpt-4-0613 for adapting the HotpotQA insights into FEVER insights. We use the same fewshot examples to finetune the insights as the ones that will be used during task execution.

在这个实验中，我们将 HotpotQA 数据集作为源任务，将 FEVER 数据集作为目标任务。 与 HotpotQA 数据集一样，我们赋予智能体使用 Docstore API 浏览 Wikipedia 的能力；因此，我们假设从 HotpotQA 任务获得的部分知识在迁移到 FEVER 任务时也应当有益。 我们使用 gpt-4-0613 将 HotpotQA 洞见适配为 FEVER 洞见。 我们使用与任务执行期间相同的少样本示例来微调洞见。

We compare our ExpeL Transfer agent's transfer learning ability with (1) ReAct; (2) Act; and (3) an agent that "finetunes" insights without task demonstrations. Notice that since source and target tasks are inherently different, we do not have an experience pool to retrieve from; thus, the ExpeL Transfer agents use the existing fixed fewshot examples as in-context examples.

我们将 ExpeL Transfer 智能体的迁移学习能力与以下方法比较：（1）ReAct；（2）Act；（3）在没有任务演示的情况下“微调”洞见的智能体。 需要注意的是，由于源任务与目标任务本质不同，我们没有可供检索的经验池；因此，ExpeL Transfer 智能体使用现有固定少样本示例作为上下文示例。

Table 1 showcases the transfer learning results. Both agents that transferred knowledge from the source domain saw performance gains. Notably, the agent with a few in-context examples had a more significant improvement than the one without, indicating the effectiveness of the proposed "finetuning" method in transfer learning scenarios.

表1 展示了迁移学习结果。 两个从源领域迁移知识的智能体都获得了性能提升。 值得注意的是，使用少量上下文示例的智能体比不使用示例的智能体提升更显著，这表明所提出的“微调”方法在迁移学习场景中有效。

表1：迁移结果。我们将从 HotpotQA 抽取的洞见迁移到 FEVER；Act 和 ReAct 是基线智能体，ExpeL Transfer w/o Task Demos 在为目标任务修改洞见时不使用少样本示例。

Method	FEVER (SR %)
Act	58 ± 0.0
ReAct	63 ± 0.4
ExpeL Transfer w/o Task Demos	65 ± 1.7
ExpeL Transfer	70 ± 0.7

表2：ALFWorld 上使用 Reflexion 轮次的成功率。ExpeL 与 Reflexion 在 ALFWorld 环境中表现出协同作用；R0 高亮表示 ExpeL 只尝试一次，R1–R3 从失败的 R0 检查点继续。

Method	R0	R1	R2	R3
ReAct+Reflexion	40.3%	47.8%	52.2%	54.4%
ExpeL retrieve only	54.5%	57.5%	59.7%	60.4%
ExpeL+Reflexion	59.0%	60.4%	63.4%	64.2%

5.5 ExpeL with Task Reattempts

While not being the central focus of our study, we present preliminary findings on the effectiveness of incorporating task reattempts into the evaluation phase using ExpeL by resuming the failed checkpoints from R0. The performance of ExpeL combined with Reflexion, alongside two baselines: ReAct/Reflexion and ExpeL without insights (ExpeL retrieve only), is detailed in Table 2. The results demonstrate a notable improvement in the success rate when ExpeL is paired with Reflexion, with the success rate increasing as the number of task reattempts grows.

尽管这不是本研究的核心重点，我们仍给出了一些初步发现：在评估阶段使用 ExpeL，并从失败的 R0 检查点恢复以重新尝试任务，能够带来怎样的效果。 ExpeL 与 Reflexion 结合后的性能，以及 ReAct/Reflexion 和不使用洞见的 ExpeL（仅检索 ExpeL）两个基线的性能，详见 表2。 结果表明，将 ExpeL 与 Reflexion 结合会显著提高成功率，并且随着任务重试次数增加，成功率继续提升。

5.6 Ablation Studies

One main component of ExpeL is the agent's ability to autonomously gather valuable experiences benefiting its own learning. Therefore, we wish to investigate if the number of useful experiences impacts the downstream performance of ExpeL. We designed two different agents to compare our agent with. The first one only has access to initial fewshot examples and extracts insights from them. The second gathers experience using ReAct where the agent has no retries. Thus, the agent will not only get less successful trajectories but will also not have any success/failure comparison pairs during insights extraction.

ExpeL 的一个主要组件，是智能体自主收集有助于自身学习的宝贵经验的能力。 因此，我们希望研究有用经验的数量是否会影响 ExpeL 的下游性能。 我们设计了两个不同的智能体与我们的方法进行比较。 第一个智能体只能访问初始少样本示例，并从中抽取洞见。 第二个智能体使用 ReAct 收集经验，且不允许重试。 因此，该智能体不仅获得的成功轨迹更少，在洞见抽取期间也没有任何成功/失败比较对。

We conducted experiments in the HotpotQA environment and presented the results in Figure 6. As we can see, the agent that extracts insights from the existing fewshots has no advantage compared to the ReAct agent, illustrating that experience is essential for ExpeL to learn from. This was reflected in a significantly better performance for the two other agents having access to more experience. Furthermore, the ExpeL agent with access to a diverse set of experiences (failure and success pairs obtained using Reflexion) performs better than the agent using only ReAct during experience gathering.

我们在 HotpotQA 环境中进行了实验，结果见 图6。 可以看到，仅从已有少样本示例中抽取洞见的智能体相对于 ReAct 智能体没有优势，这说明经验是 ExpeL 学习所必需的。 另两个能够访问更多经验的智能体性能显著更好，也反映了这一点。 此外，能够访问多样化经验集合（使用 Reflexion 获得的失败与成功经验对）的 ExpeL 智能体，优于经验收集期间只使用 ReAct 的智能体。

图6：经验对性能的影响。图中突出多样化经验样本数量与最终性能之间的相关性，比较 ReAct、只能访问少样本示例的 ExpeL，以及经验收集阶段只使用 ReAct 的 ExpeL。额外的自主收集经验对 ExpeL 的成功至关重要，使用 Reflexion 收集的成功/失败数据多样性也优于只使用 ReAct。

Next, we will scrutinize the efficacy of the insight extraction step of ExpeL. Since insights had the most significant impact on the HotpotQA environment (Figure 5), we performed the ablations on insights in this environment. We use three dimensions to ablate the design choices for insight extraction by creating the following variants of ExpeL agents: (1) human-crafted insights (Figure in Appendix), which were manually engineered by carefully studying the agent's mistakes during the experience gathering step; (2) adding reflections $\nu$ into the insights construction step in addition to using fail/success pairs and lists of successes; (3) using gpt-3.5-turbo-0613 as the ${\text{LLM}}_{\text{insights}}$.

接下来，我们仔细考察 ExpeL 洞见抽取步骤的效果。 由于洞见在 HotpotQA 环境中影响最大（图5），我们在该环境中对洞见进行消融。 我们从三个维度消融洞见抽取的设计选择，并创建以下 ExpeL 变体：（1）人工编写的洞见（见附录中的图），通过仔细研究智能体在经验收集阶段的错误手工设计；（2）除了使用失败/成功经验对和成功经验列表之外，还在洞见构建步骤中加入反思 $\nu$；（3）使用 gpt-3.5-turbo-0613 作为 ${\text{LLM}}_{\text{insights}}$。

Results in Table 3 show several significant findings: (1) learned insights by the agent are more advantageous than hand-crafted ones; (2) using reflections in addition to success/failure pairs and lists of successes is disadvantageous, possibly due to reflections sometimes outputting hallucinations, therefore misleading the insight extraction stage; and (3) a better LLM is more advantageous at improving ExpeL's performance, suggesting our agent will enjoy free performance boosts with the ever-improving nature of base foundation models.

表3 的结果揭示了几个重要发现：（1）智能体学得的洞见比人工编写的洞见更有优势；（2）在成功/失败经验对和成功经验列表之外再使用反思，反而不利，这可能是因为反思有时会产生幻觉，从而误导洞见抽取阶段；（3）更好的 LLM 更有利于提升 ExpeL 性能，这表明随着基础模型持续改进，我们的智能体可以自然获得性能提升。

Lastly, we investigated the design choice of using task similarity as the ranking score for retrieving successful in-context examples in ALFWorld. In particular, we use (1) reason similarity by retrieving top-$k$ trajectories with the most similar reasoning step as the latest reasoning step in the current trajectory, and (2) randomly sampling successful trajectories from the experience pool. We clearly observe in Table 3 that retrieving with task similarity (ExpeL) performs the best. Reason similarity is still advantageous but slightly drops in performance, possibly due to dynamically changing fewshots during a single trajectory, causing instabilities. Lastly, random sampling has a significant drop in performance, suggesting that our design choice of selecting the most pertinent in-context example is advantageous.

最后，我们研究了在 ALFWorld 中使用任务相似性作为成功上下文示例检索排序分数的设计选择。 具体而言，我们使用：（1）推理相似性，检索其推理步骤与当前轨迹最新推理步骤最相似的 top-$k$ 条轨迹；（2）从经验池中随机采样成功轨迹。 从 表3 可以清楚看到，使用任务相似性进行检索的 ExpeL 表现最好。 推理相似性仍然有益，但性能略有下降，这可能是因为少样本示例在单条轨迹中动态变化，导致不稳定。 最后，随机采样的性能显著下降，说明我们选择最相关上下文示例的设计是有益的。

表3：消融结果。上：洞见抽取消融。人工编写的洞见优于 ReAct，但不如 LLM 生成的洞见；加入反思会损害性能，而更好的基础 LLM 会生成更好的洞见。下：上下文示例选择策略消融。随机选择会导致性能显著下降，按推理相似性排序也有明显下降。

Variant	Success Rate (%)
HotpotQA: insight extraction
ReAct	28.0 ± 1.4
Hand-crafted insights	32.0 ± 1.1
Insights with reflections	29.0 ± 0.4
gpt-3.5-turbo insights	32.0 ± 0.4
ExpeL (ours)	39.0 ± 1.7
ALFWorld: in-context example selection
ReAct	40.0 ± 0.3
Reasoning similarity	48.5 ± 2.1
Random sampled	42.5 ± 0.8
ExpeL (ours)	59.0 ± 0.3

6. Conclusion and Limitations

Limitations. In this work, we investigated tasks with textual observation, which is limiting in real-world scenarios. Thus, incorporating image observations will make our method more generally applicable. Using Vision-Language Models or captioning models to supplement the LLM to enable image observations could be an interesting new avenue of research. Additionally, we investigated the efficacy of our method by using closed-source API LLMs, which can be off-limits in some applications. Exploring LLM agents using open-source LLMs should be another promising future work.

局限性。 本文研究的是具有文本观察的任务，这在现实场景中存在局限。 因此，引入图像观察将使我们的方法具备更广泛的适用性。 使用视觉语言模型或图像描述模型来补充 LLM，使其能够处理图像观察，可能是一个有趣的新研究方向。 此外，我们使用闭源 API LLM 来研究方法效果，而某些应用可能无法使用这些模型。 探索使用开源 LLM 的 LLM 智能体，应当是另一个很有前景的未来方向。

Furthermore, since our extracted insights do not exceed the current LLM's token limit, we can fit them into the agent's context window. However, extra retrieval steps for insights might be needed for truly lifelong learning agents to ensure a manageable context window size. Lastly, unlike reinforcement learning methods, prompting techniques lack theoretical underpinnings that could potentially impact the efficiency of the resulting policies. Future research should explore the integration of these approaches to yield more effective and optimal solutions.

此外，由于我们抽取的洞见不超过当前 LLM 的 token 限制，因此可以将它们放入智能体的上下文窗口。 然而，对于真正的终身学习智能体，可能需要增加洞见检索步骤，以确保上下文窗口大小可控。 最后，与强化学习方法不同，提示技术缺少理论基础，这可能影响所得策略的效率。 未来研究应探索这些方法的融合，以获得更有效、更优的解决方案。

In summary, we introduced ExpeL, a novel learning LLM agent that autonomously gathers experience from a set of training tasks to improve its abilities in solving evaluation tasks without access to model parameters. We demonstrated its learning abilities by showing its performance gain compared to vanilla ReAct and Act agents. Furthermore, we investigated a transfer learning scenario where extracting insights from a set of source tasks can benefit the ExpeL agent in solving a target task. Lastly, we presented several unexpected emerged abilities our agent developed at the end of its training. We believe that autonomously learning from experience is essential for developing human-like intelligent agents, and our ExpeL agent is a step toward that goal.

总之，我们提出 ExpeL，这是一种新型学习型 LLM 智能体；它从一组训练任务中自主收集经验，无需访问模型参数，就能提高解决评估任务的能力。 通过展示它相对于原始 ReAct 和 Act 智能体的性能提升，我们证明了其学习能力。 此外，我们研究了一个迁移学习场景，其中从一组源任务抽取洞见，可以帮助 ExpeL 智能体解决目标任务。 最后，我们展示了智能体在训练结束时形成的几种意外涌现能力。 我们相信，自主地从经验中学习对于开发类人智能体至关重要，而 ExpeL 智能体正是朝这一目标迈出的一步。