理解 LLM 智能体的规划：综述
Abstract
As Large Language Models (LLMs) have shown significant intelligence, the progress to leverage LLMs as planning modules of autonomous agents has attracted more attention. This survey provides the first systematic view of LLM-based agents planning, covering recent works aiming to improve planning ability. We provide a taxonomy of existing works on LLM-Agent planning, which can be categorized into Task Decomposition, Plan Selection, External Module, Reflection and Memory. Comprehensive analyses are conducted for each direction, and further challenges for the field of research are discussed.
随着大语言模型（LLM）展现出显著的智能，利用 LLM 作为自主智能体规划模块的研究进展受到了更多关注。 本综述首次系统梳理基于 LLM 的智能体规划，涵盖旨在提升规划能力的近期工作。 我们对现有 LLM 智能体规划工作进行分类，将其划分为 任务分解、计划选择、外部模块、反思 和 记忆。 我们对每个方向进行了全面分析，并讨论了该研究领域面临的进一步挑战。
1. Introduction
Autonomous agents have been recognized as intelligent entities capable of accomplishing specific tasks, via perceiving the environment, planning, and executing actions. Planning, as one of the most critical capabilities for agents, requires complicated understanding, reasoning, and decision-making progress.
自主智能体被视为能够通过 感知环境、规划 和 执行动作 来完成特定任务的智能实体。 规划是智能体最关键的能力之一，需要复杂的理解、推理与决策过程。
Despite the abstract concept of planning, a general formulation of the planning tasks can be described as follows. Given time step
尽管规划是一个抽象概念，但规划任务可以用如下通用形式来描述。 给定时间步
where
其中，
Conventional works mainly rely on symbolic methods or reinforcement learning-based methods, such as Planning Domain Definition Language (PDDL) or policy learning. However, those conventional methods have several limitations. Symbolic methods require conversion from flexible natural language-described problems into symbolic modeling, which may require human experts' efforts. Usually, this kind of method lacks error tolerance, resulting in failures even if there are only a few errors.
传统工作主要依赖符号方法或基于强化学习的方法，例如规划领域定义语言（PDDL）或策略学习。 然而，这些传统方法存在若干局限。 符号方法需要把以灵活自然语言描述的问题转换为符号模型，这可能需要人类专家投入精力。 这类方法通常缺乏容错能力，即使只有少量错误也会导致失败。
Reinforcement learning (RL) methods are often combined with deep models, which serve as the policy network or reward model. While RL algorithms often require a large number of samples (interactions with the environment) to learn an effective policy, this can be impractical or costly in scenarios where collecting data is time-consuming or expensive.
强化学习（RL）方法通常与深度模型结合，后者充当策略网络或奖励模型。 强化学习算法往往需要大量样本（即与环境的交互）才能学到有效策略；在数据收集耗时或成本高昂的场景中，这可能并不现实或代价过高。
In recent years, the emergence of Large Language Models (LLMs) has marked a paradigm shift. LLMs have achieved remarkable success across various domains, showcasing significant intelligence in reasoning, tool usage, planning, and instruction-following. The surprising intelligence of LLMs sheds light on employing LLMs as the cognitive core of agents, thereby offering the potential to improve planning ability. Numerous methodologies have been developed to harness the potential of LLMs for agent planning.
近年来，大语言模型（LLM）的出现标志着一次范式转变。 LLM 在多个领域取得了显著成功，在推理、工具使用、规划和指令遵循方面展现出强大智能。 LLM 令人惊讶的智能启发人们将其用作智能体的认知核心，从而为提升规划能力带来潜力。 研究者已经提出许多方法，以发挥 LLM 在智能体规划方面的潜力。
While existing surveys have attempted to summarize techniques for LLMs, LLMs for decision-making, reasoning, tool learning, and autonomous agents, they often lack a detailed analysis of planning ability within the literature. In this survey, we analyze the latest research works and discuss the advantages and limitations, aiming to provide a systematic view of the planning ability of LLM-based agents. Existing methods are further categorized into five representative directions, with each direction undergoing comprehensive analysis. Furthermore, we have evaluated several representative methods on four benchmarks. To the best of our knowledge, this is the first work that comprehensively analyzes LLM-based agents from the planning abilities.
尽管现有综述尝试总结 LLM 技术、用于决策的 LLM、推理、工具学习和自主智能体，但这些文献往往缺少对规划能力的详细分析。 在本综述中，我们分析最新研究工作并讨论其优势与局限，旨在系统呈现基于 LLM 的智能体所具备的规划能力。 我们进一步将现有方法归纳为五个代表性方向，并对每个方向进行全面分析。 此外，我们还在四个基准上评估了若干代表性方法。 据我们所知，这是首项从规划能力角度全面分析基于 LLM 的智能体的工作。
The subsequent sections of this paper are organized as follows. In Section 2, we categorize the works into five mainstream directions and analyze their ideas regarding planning ability. Sections 3 to 7 provide detailed discussions and analysis of each direction. Finally, Section 9 concludes the survey, offering insights into future directions in this field.
本文后续章节安排如下。 在第 2 节中，我们将相关工作划分为五个主流方向，并分析它们关于规划能力的核心思想。 第 3 至第 7 节分别对各方向进行详细讨论与分析。 最后，第 9 节总结全文，并展望该领域的未来方向。
2. Taxonomy
As the research on the planning ability of LLM-based agents presents a flourishing scene, various methods have been proposed to exploit the upper limit of planning ability. To have a better bird's view of existing advanced works, we pick out some representative and influential works, analyzing their motivations and essential ideas. To provide a better understanding, we illustrate the analysis in Table 1.
随着基于 LLM 的智能体规划能力研究蓬勃发展，研究者提出了多种方法来探索规划能力的上限。 为了更全面地俯瞰现有先进工作，我们选取了一些具有代表性和影响力的研究，分析其动机与核心思想。 为了便于理解，我们在表1中展示了这一分析。
According to the table, we present a novel and systematic taxonomy for LLM-based agent plannning that divides existing works into five important categories, covering task decomposition, multi-plan selection, external module-aided planning, reflection and refinement and memory-augmented planning, as illustrated in Figure 1. Here we briefly summarize those five directions as below.
根据该表，我们提出一种新颖且系统的 LLM 智能体规划分类体系，将现有工作划分为五个重要类别：任务分解、多计划选择、外部模块辅助规划、反思与改进 以及 记忆增强规划，如图1所示。 下面简要概述这五个方向。
Task Decomposition. Tasks in real life are usually complicated and multi-step, bringing severe hardness for planning. This kind of method adopts the idea of divide and conquer, decomposing the complicated into several sub-tasks and then sequentially planning for each sub-task. The process could be formulated as follows:
任务分解。 现实生活中的任务通常复杂且包含多个步骤，给规划带来很大困难。 这类方法采用 分而治之 的思想，将复杂任务分解为若干子任务，再依次为每个子任务进行规划。 该过程可形式化为：
Multi-plan Selection. This kind of method focuses on leading the LLM to ``think'' more, generating various alternative plans for a task. Then a task-related search algorithm is employed to select one plan to execute. The process could be formulated as follows:
多计划选择。 这类方法着重引导 LLM 进行更多“思考”，为任务生成多种候选计划。 随后使用与任务相关的搜索算法选择一个计划执行。 该过程可形式化为：
where
其中，
External Planner-Aided Planning. This methodology is crafted to employ an external planner to elevate the planning procedure, aiming to address the issues of efficiency and infeasibility of generated plans, while the LLM mainly plays the role in formalizing the tasks. The process could be formulated as follows:
外部规划器辅助规划。 这类方法使用外部规划器来增强规划过程，旨在解决生成计划效率低或不可行的问题，而 LLM 主要负责将任务形式化。 该过程可形式化为：
where
其中，
Reflection and Refinement. This methodology emphasizes improving planning ability through reflection and refinement. It encourages LLM to reflect on failures and then refine the plan. The process could be formulated as follows:
反思与改进。 这类方法强调通过反思与改进来提升规划能力。 它鼓励 LLM 反思失败，再改进计划。 该过程可形式化为：
Memory-augmented Planning. This kind of approach enhances planning with an extra memory module, in which valuable information is stored, such as commonsense knowledge, past experiences, domain-specific knowledge, et al. The information is retrieved when planning, serving as auxiliary signals. The process could be formulated as follows:
记忆增强规划。 这类方法借助额外的记忆模块增强规划，并在其中存储有价值的信息，例如常识知识、过往经验和领域特定知识等。 规划时会检索这些信息，将其作为辅助信号。 该过程可形式化为：
where
其中，
The five directions are interconnected rather than mutually exclusive, often involving the concurrent adoption of multiple techniques. In the subsequent sections, we delve deeper into the five research directions concerning LLM-agent planning, elucidating their motivations, proposing representation solutions, and addressing inherent limitations.
这五个方向相互关联而非彼此排斥，往往会同时采用多种技术。 在后续章节中，我们将深入讨论 LLM 智能体规划的五个研究方向，阐明其动机、介绍代表性解决方案并分析其固有局限。
| Method | Idea | LLM's task | Formulation | Representative works |
|---|---|---|---|---|
| Task Decomposition | Divide and Conquer | Task decomposition Subtask planning | CoT, ReAct, HuggingGPT | |
| Multi-plan Selection | Generate multiple plans and select the optimal | Plans generation Plans evaluation | ToT, GoT, CoT-SC | |
| External Planner-aided | Formalize tasks and utilize external planner | Task formalization | LLM+P, LLM+PDDL | |
| Reflection & Refinement | Reflect on experiences and refine plans | Plan generation Reflection Refinement | Reflexion, CRITIC, Self-Refine | |
| Memory-aided Planning | Leverage memory to aid planning | Plan generation Memory extraction | REMEMBER, MemoryBank |
3. Task Decomposition
In real-world scenarios, environments are often characterized by complexity and variability, thereby addressing complex tasks through a one-step planning process is a formidable challenge.
在现实场景中，环境往往具有复杂性和可变性，因此通过一步规划过程解决复杂任务是一项艰巨挑战。
This simplification of complicated tasks is a remarkable human ability, evident in the decomposition of one task into several simpler sub-tasks, which is analogous to the well-known algorithmic strategy called ``divide and conquer", as illustrated in Eq. (1). Task decomposition generally involves two crucial steps: firstly, decomposing the complex task, referred to as the decompose" step, and secondly, planning for the sub-tasks, known as the sub-plan step". Current methods for task decomposition in this domain generally fall into two categories: decomposition-first and interleaved decomposition, illustrated in Figure 2.
这种简化复杂任务的能力是人类的一项卓越能力，体现为将一个任务分解成若干更简单的子任务；它类似于公式（1）所示、广为人知的“分而治之”算法策略。 任务分解通常包含两个关键步骤：首先分解复杂任务，即“分解”步骤；其次为子任务进行规划，即“子计划”步骤。 该领域当前的任务分解方法通常分为两类：先分解方法和交错分解方法，如图2所示。
3.1 Decomposition-First Methods
Decomposition-first methods decompose the task into sub-goals first and then plan for each sub-goal successively, presented in Figure 2(a). The representative methods include HuggingGPT, Plan-and-Solve, ProgPrompt, et al.
先分解方法首先将任务分解为若干子目标，再依次为每个子目标进行规划，如图2（a）所示。 代表性方法包括 HuggingGPT、Plan-and-Solve 和 ProgPrompt 等。
HuggingGPT utilizes various multimodal models from the Huggingface Hub to construct an intelligent agent for multimodal tasks. It is capable of handling tasks such as image generation, image classification, object recognition, video annotation, speech-to-text, et al. To facilitate collaboration between different models, the LLM acts as a controller, responsible for decomposing tasks inputted by humans, selecting models, and generating final responses. The most crucial stage is the initial task decomposition, where HuggingGPT explicitly instructs the LLM to break down the given task into sub-tasks, providing dependencies between tasks.
HuggingGPT 利用 Hugging Face Hub 中的多种多模态模型构建用于多模态任务的智能体。 它能够处理图像生成、图像分类、目标识别、视频标注和语音转文本等任务。 为了促进不同模型之间的协作，LLM 充当控制器，负责分解人类输入的任务、选择模型并生成最终响应。 其中最关键的阶段是初始任务分解：HuggingGPT 明确指示 LLM 将给定任务拆分为子任务，并给出任务之间的依赖关系。
Plan-and-Solve improves upon the Zero-shot Chain-of-Thought by transforming the original *Let's think step-by-step*" into a two-step prompt instruction: Let's first devise a plan" and ``Let's carry out the plan". This zero-shot approach has achieved improvements in mathematical reasoning, common-sense reasoning, and symbolic reasoning.
Plan-and-Solve 改进了零样本思维链，将原本的“让我们逐步思考”改为两步提示指令：“让我们先制定计划”和“让我们执行计划”。 这种零样本方法在数学推理、常识推理和符号推理方面均取得了提升。
ProgPrompt translates natural language descriptions of tasks into coding problems. It symbolizes the agent's action space and objects in the environment through code, with each action formalized as a function and each object represented as a variable. Consequently, task planning is naturally transformed into function generation. When executing tasks, the agent first generates a plan in the form of function callings and then executes them step by step.
ProgPrompt 将任务的自然语言描述转换为编程问题。 它用代码将智能体的动作空间与环境中的对象符号化：每个动作被形式化为函数，每个对象被表示为变量。 因此，任务规划自然转化为函数生成。 执行任务时，智能体首先以函数调用的形式生成计划，再逐步执行这些调用。
3.2 Interleaved Decomposition Methods
Interleaved decomposition involves interleaved task decomposition and sub-task planning, where each decomposition only reveals one or two sub-tasks at the current state, illustrated in Figure 2(b). Representative methods in this category include the Chain-of-Thought (CoT) series, ReAct, PAL, Program-of-Thought (PoT), Visual ChatGPT, et al.
交错分解将任务分解与子任务规划交替进行，每次分解在当前状态下只给出一到两个子任务，如图2（b）所示。 这一类别的代表性方法包括思维链（CoT）系列、ReAct、PAL、程序思维（PoT）和 Visual ChatGPT 等。
The introduction of Chain-of-Thought (CoT) reveals the few-shot learning capabilities of LLM. CoT guides the LLM in reasoning about complex problems through a few constructed trajectories, leveraging the LLM's reasoning abilities for task decomposition. Subsequently, Zero-shot CoT unlocks the LLM's zero-shot reasoning abilities with the magical instruction ``Let's think step-by-step".
思维链（CoT）的提出揭示了 LLM 的少样本学习能力。 CoT 通过少量构造的轨迹引导 LLM 推理复杂问题，利用 LLM 的推理能力进行任务分解。 随后，零样本 CoT 使用“让我们逐步思考”这一神奇指令，激发 LLM 的零样本推理能力。
In contrast to CoT, which embeds reasoning within the planning process, ReAct decouples reasoning and planning. It alternates between reasoning (Thought step) and planning (Action step), demonstrating significant improvements in the planning capabilities. Visual ChatGPT utilizes ReAct's mechanism, employing LLM as the agent's brain equipped with a series of visual models, resulting in an agent with image processing capabilities.
CoT 将推理嵌入规划过程，而 ReAct 与之不同，它将推理与规划解耦。 ReAct 在推理（Thought 步骤）与规划（Action 步骤）之间交替，显著提升了规划能力。 Visual ChatGPT 采用 ReAct 的机制，把 LLM 用作智能体的大脑，并为其配备一系列视觉模型，从而形成具备图像处理能力的智能体。
PAL improves CoT by leveraging the LLM's coding abilities, guiding the LLM to generate code during reasoning. Finally, a code interpreter (such as Python) is used to comprehensively execute the codes to obtain the solution. This method proves helpful for agents in solving mathematical and symbolic reasoning problems. Program-of-Thought (PoT) completely formalize the reasoning process as programming. The authors also leverage a CodeX model trained on code-related data, enhancing performance in mathematical and financial problems.
PAL 利用 LLM 的编程能力改进 CoT，引导 LLM 在推理过程中生成代码。 最后，代码解释器（如 Python）会完整执行这些代码并得到答案。 事实证明，该方法有助于智能体解决数学和符号推理问题。 程序思维（PoT）将推理过程完全形式化为编程。 我们还采用了一个在代码相关数据上训练的 CodeX 模型，从而提升在数学和金融问题上的表现。
3.3 Discussions
For the decomposition-first method, the advantage lies in creating a stronger correlation between the sub-tasks and the original tasks, reducing the risk of task forgetting and hallucinations. However, since the sub-tasks are predetermined at the beginning, additional mechanisms for adjustment are required otherwise one error in some step will result in failure, which will be discussed in Section 6.
先分解方法的优势在于，它能在子任务与原始任务之间建立更强的关联，降低任务遗忘和幻觉的风险。 然而，由于子任务在一开始就已预先确定，因此还需要额外的调整机制，否则某一步出现错误就会导致失败；第 6 节将进一步讨论这一问题。
On the other hand, interleaved decomposition and sub-planning dynamically adjust decomposition based on environmental feedback, improving the fault tolerance. However, for complicated tasks, excessively long trajectories may lead to LLM experiencing hallucinations, deviating from the original goals during subsequent sub-tasks and sub-planning.
另一方面，交错分解与子规划会根据环境反馈动态调整分解过程，从而提高容错能力。 但是，对于复杂任务，过长的轨迹可能导致 LLM 产生幻觉，并在后续子任务和子规划过程中偏离原始目标。
Although task decomposition significantly enhances the ability of LLM-Agent to solve complicated tasks, challenges persist. The first challenge is the additional overhead introduced by task decomposition. Decomposing a task into multiple sub-tasks requires more reasoning and generation, incurring additional time and computational costs. On the other hand, for highly complex tasks that are decomposed into dozens of sub-tasks, the planning is constrained by the context length of the LLM, leading to the forgetting of the planning trajectories.
尽管任务分解显著增强了 LLM 智能体解决复杂任务的能力，但挑战依然存在。 第一项挑战是任务分解带来的额外开销。 将一个任务分解为多个子任务需要更多推理和生成，因此会增加时间与计算成本。 另一方面，对于被分解为数十个子任务的高度复杂任务，规划会受到 LLM 上下文长度的限制，进而遗忘规划轨迹。
4. Multi-Plan Selection
Due to the complexity of the tasks and the inherent uncertainty of LLM, the plans generated by the LLM-Agent for a given task can be diverse. Even though LLM possesses strong reasoning abilities, a single plan generated by LLM is likely to be suboptimal or even infeasible. A more natural approach is multi-plan selection, comprising two major steps: multi-plan generation and optimal plan selection.
由于任务的复杂性和 LLM 固有的不确定性，LLM 智能体针对同一任务生成的计划可能多种多样。 即使 LLM 具有很强的推理能力，它生成的单个计划也可能并非最优，甚至不可行。 一种更自然的方法是多计划选择，它包含两个主要步骤：多计划生成和最优计划选择。
4.1 Multi-Plan Generation
Multi-plan generation involves generating a dozen paths of plans to comprise the candidate plan set. Mainstream methods consider employing uncertainty in the decoding process of generative models.
多计划生成会产生十余条计划路径，以组成候选计划集合。 主流方法考虑利用生成模型解码过程中的不确定性。
Self-consistency employs a simple intuition: the solutions for complex problems are rarely unique. In contrast to CoT, which generates a single path, Self-consistency obtains multiple distinct reasoning paths via sampling strategies embodied in the decoding process, such as temperature sampling, top-k sampling.
Self-consistency 采用一个简单直觉：复杂问题的解很少是唯一的。 CoT 只生成一条路径，而 Self-consistency 通过解码过程中采用的采样策略获得多条不同推理路径，例如温度采样和 top-k 采样。
Tree-of-Thought (ToT) proposes two strategies to generate plans (i.e. thoughts): sample and propose. The sample strategy is consistent with Self-consistency, where LLM would sample multiple plans in decoding process. The propose strategy explicitly instructs the LLM to generate various plans via few-shot examples in prompts. Graph-of-Thought (GoT) extends ToT by adding transformations of thoughts, which supports arbitrary thoughts aggregation.
思维树（ToT）提出两种生成计划（即思维）的策略：采样和提议。 采样 策略与 Self-consistency 一致，LLM 会在解码过程中采样多个计划。 提议 策略则通过提示中的少样本示例，明确要求 LLM 生成多种计划。 思维图（GoT）通过加入思维变换来扩展 ToT，从而支持任意思维聚合。
LLM-MCTS and RAP leverages LLM as the heuristic policy function for the Monte Carlo Tree Search (MCTS), where multiple potential actions are obtained by multiple calls.
LLM-MCTS 和 RAP 将 LLM 用作蒙特卡洛树搜索（MCTS）的启发式策略函数，通过多次调用获得多个潜在动作。
4.2 Optimal Plan Selection
To select the optimal plan among the candidate plans, diverse strategies are adopted as heuristic search algorithms.
为了从候选计划中选出最优计划，研究者采用多种策略作为启发式搜索算法。
Self-consistency applies the naive majority vote strategy, regarding the plan with the most votes as the optimal choice. Benefiting from the tree architecture, Tree-of-Thought (ToT) supports tree search algorithms, such as conventional BFS and DFS. When selecting a node for expansion, it uses LLM to evaluate multiple actions and chooses the optimal one.
Self-consistency 采用朴素多数投票策略，把得票最多的计划视为最优选择。 得益于树结构，思维树（ToT）支持传统的广度优先搜索（BFS）和深度优先搜索（DFS）等树搜索算法。 选择待扩展节点时，它使用 LLM 评估多个动作并选出最优动作。
Similar with ToT, LLM-MCTS and RAP also employ a tree structure to assist in multi-plan search. Unlike ToT, they employ the Monte Carlo Tree Search (MCTS) algorithm for search. LLM A* utilizes the classic A* algorithm from artificial intelligence to assist LLM in search. The Chebyshev distance from the current position to the target position serves as the heuristic cost function for selecting the optimal path.
与 ToT 类似，LLM-MCTS 和 RAP 也使用树结构辅助多计划搜索。 不同于 ToT，它们采用蒙特卡洛树搜索（MCTS）算法进行搜索。 LLM A* 使用人工智能中的经典 A* 算法辅助 LLM 搜索。 从当前位置到目标位置的切比雪夫距离被用作选择最优路径的启发式代价函数。
4.3 Discussions
The scalability of multi-plan selection is notably advantageous, providing a broader exploration of potential solutions in the expansive search space. However, this advantage comes with inherent trade-offs. The increased computational demands, especially for models with large token counts or computations, pose practical challenges. This cost consideration becomes crucial, particularly in scenarios where resource constraints are a significant factor, such as the online service.
多计划选择具有显著的可扩展性优势，可以在广阔的搜索空间中更充分地探索潜在解。 然而，这项优势伴随着固有权衡。 计算需求的增加会带来实际挑战，尤其是对于 token 数量或计算量很大的模型。 在资源限制显著的场景中，例如在线服务，这项成本因素尤其关键。
Moreover, the reliance on LLM for the evaluation of plans introduces new challenges. As LLM's performance in ranking tasks is still under scrutiny, there is a need for further validation and fine-tuning of its capabilities in this specific context. The stochastic nature of LLMs adds randomness to the selection, potentially affecting the consistency and reliability of the chosen plans.
此外，依赖 LLM 评估计划会引入新的挑战。 LLM 在排序任务上的表现仍有待检验，因此需要进一步验证并针对这一特定场景微调其能力。 LLM 的随机性会给选择过程增加不确定因素，可能影响所选计划的一致性和可靠性。
5. External Planner-Aided Planning
Despite the powerful reasoning and task decomposition capabilities exhibited by Large Language Models (LLMs), challenges arise when confronted with environments featuring intricate constraints, such as mathematical problem-solving or generating admissible actions. To address challenges, several methods integrate LLMs with external planners. Such methods can be categorized into symbolic planners and neural planners based on the introduced planners.
尽管大语言模型（LLM）展现出强大的推理和任务分解能力，但当面对具有复杂约束的环境时仍会遇到挑战，例如求解数学问题或生成合法动作。 为了应对这些挑战，一些方法将 LLM 与外部规划器结合。 根据所引入规划器的类型，这些方法可分为符号规划器和神经规划器。
5.1 Symbolic Planner
Symbolic planners have served as a fundamental component in the fields of automated planning for several decades. These approaches, based on well-established symbolic formalized models, such as PDDL models, employ symbolic reasoning to identify optimal paths from initial states to desired goal states.
数十年来，符号规划器一直是自动规划领域的基础组件。 这些方法基于 PDDL 模型等成熟的符号形式化模型，使用符号推理寻找从初始状态到期望目标状态的最优路径。
LLM+P enhances the planning proficiency of LLMs by incorporating a PDDL-based symbolic planner. Leveraging the semantic understanding and coding capabilities of LLM, the authors organize problems into textual language prompts inputted to LLM. This prompts LLM to organize the actions within the environment and specified tasks into the format of the PDDL language. Subsequently, after obtaining a formalized description, the authors employ the Fast-Downward solver for the planning process.
LLM+P 通过引入基于 PDDL 的符号规划器来增强 LLM 的规划能力。 我们利用 LLM 的语义理解和编程能力，将问题整理为文本语言提示并输入 LLM。 这会促使 LLM 将环境中的动作和指定任务组织为 PDDL 语言格式。 随后，在获得形式化描述后，我们使用 Fast-Downward 求解器执行规划过程。
Building upon LLM+P, LLM-DP is specifically designed for dynamic interactive environments. Upon receiving feedback from the environment, LLM processes the information, formalizes it into PDDL language, and then employs a BFS solver to generate a plan.
LLM-DP 在 LLM+P 的基础上构建，专门面向动态交互环境。 收到环境反馈后，LLM 会处理这些信息，将其形式化为 PDDL 语言，再使用 BFS 求解器生成计划。
LLM+PDDL also utilizes the PDDL language to formalize the task, incorporating an additional step for manual verification to check for potential issues in the PDDL model generated by LLM. During the planning process, the authors propose using the plan generated by LLM as an initial heuristic solution to accelerate the search process of local search planners, such as LPG. LLM+ASP transforms problems described in natural language by LLM into atomic facts, converting tasks into ASP problems. Subsequently, the ASP solver CLINGO is utilized to generate plans.
LLM+PDDL 也使用 PDDL 语言将任务形式化，并加入额外的人工验证步骤，以检查 LLM 生成的 PDDL 模型是否存在潜在问题。 在规划过程中，我们提出将 LLM 生成的计划用作初始启发式解，以加速 LPG 等局部搜索规划器的搜索过程。 LLM+ASP 将 LLM 以自然语言描述的问题转化为原子事实，从而把任务转换为 ASP 问题。 随后使用 ASP 求解器 CLINGO 生成计划。
5.2 Neural Planner
Neural planners are deep models trained on collected planning data with reinforcement learning or imitation learning techniques, showing effective planning abilities within the specific domain. For instance, DRRN models the planning process as a Markov Decision Process through reinforcement learning, training a policy network to obtain a deep decision model. Decision Transformer (DT) empowers a transformer model to clone human decision-making behavior with planning data.
神经规划器是使用强化学习或模仿学习技术、在收集的规划数据上训练得到的深度模型，它们在特定领域内展现出有效的规划能力。 例如，DRRN 通过强化学习把规划过程建模为马尔可夫决策过程，并训练策略网络得到深度决策模型。 决策 Transformer（DT）使用规划数据，让 Transformer 模型模仿人类决策行为。
Well-trained neural planners exhibit excellent planning capabilities within their respective domains and demonstrate superior planning efficiency due to their smaller parameter sizes. However, when faced with complex and less frequently encountered problems, where training data is scarce, these small models tend to perform poorly due to insufficient generalization ability. Therefore, several works explore combining an LLM with a light-weight neural planner, to further enhance the planning capabilities.
训练充分的神经规划器在各自领域内具有出色的规划能力，并因参数规模较小而表现出更高的规划效率。 然而，当面对训练数据稀缺、复杂且较少遇到的问题时，这些小模型往往因泛化能力不足而表现不佳。 因此，一些工作探索将 LLM 与轻量级神经规划器结合，以进一步增强规划能力。
CALM proposed an early approach that combines a language model with an RL-based neural planner. One language model processes textual environmental information, generating a set of candidate actions as priors based on the environmental information. A DRRN policy network is then employed to re-rank these candidate actions, ultimately selecting the optimal action.
CALM 较早提出一种将语言模型与基于强化学习的神经规划器结合的方法。 一个语言模型处理文本环境信息，并根据环境信息生成一组候选动作作为先验。 随后使用 DRRN 策略网络重新排序这些候选动作，最终选出最优动作。
SwiftSage leverages the dual-process theory from cognitive psychology, dividing the planning process into slow thinking and fast thinking. The slow-thinking process involves complex reasoning and rational deliberation while fast-thinking resembles an instinctive response developed through long-term training. The authors utilize a DT model, trained through imitation learning, as the fast-thinking model for rapid plan generation. When errors occur during plan execution, indicating a more complex problem, the agent switches to the slow-thinking process, where LLM engages in reasoning and planning based on the current state. This combination of fast and slow thinking has proven to be highly effective in terms of efficiency.
SwiftSage 借鉴认知心理学中的双过程理论，将规划过程分为慢思考和快思考。 慢思考过程涉及复杂推理与理性斟酌，而快思考类似于经过长期训练形成的本能反应。 我们使用通过模仿学习训练的 DT 模型作为快思考模型，以快速生成计划。 当计划执行过程中出现错误、表明问题更为复杂时，智能体会切换到慢思考过程，由 LLM 根据当前状态进行推理和规划。 事实证明，这种快慢思考结合的方式在效率方面非常有效。
5.3 Discussions
For those strategies that leverage an additional planner for assistance, LLM primarily plays a supportive role. Its main functions involve parsing textual feedback and providing additional reasoning information to assist in planning, particularly when addressing complex problems. Specifically, the enhancement of LLM's capabilities in code generation empowers the potential to deal with more general tasks for symbolic artificial intelligence.
对于借助额外规划器的策略，LLM 主要发挥辅助作用。 它的主要功能是解析文本反馈，并提供额外的推理信息来辅助规划，尤其是在处理复杂问题时。 具体而言，LLM 代码生成能力的增强，使符号人工智能具备处理更通用任务的潜力。
Actually, a significant drawback of traditional symbolic AI systems lies in the complexity and heavy reliance on human experts in constructing symbolic models, while LLM accelerates this process, facilitating faster and more optimal establishment of symbolic models. The advantages brought by symbolic systems include theoretical completeness, stability, and interpretability. The combination of statistical AI with LLM is poised to become a major trend in the future development of artificial intelligence.
事实上，传统符号人工智能系统的一项显著缺点，是构建符号模型过程复杂且高度依赖人类专家；LLM 则能加速这一过程，促进更快速、更优地建立符号模型。 符号系统带来的优势包括理论完备性、稳定性和可解释性。 统计人工智能与 LLM 的结合有望成为人工智能未来发展的重要趋势。
6. Reflection and Refinement
Reflection and refinement are indispensable components in the planning process. They enhance the fault tolerance and error correction capabilities of LLM-Agent planning. Due to existing hallucination issues and insufficient reasoning abilities for complex problems, LLM-Agents may make errors and get stuck in "thought loops" during planning due to limited feedback. Reflecting on and summarizing failures helps agents correct errors and break out of such loops in subsequent attempts.
反思与完善是规划过程中不可或缺的组成部分。 它们增强了 LLM 智能体规划的容错能力和纠错能力。 由于现有的幻觉问题以及对复杂问题推理能力不足，LLM 智能体可能会在规划中出错，并因反馈有限而陷入“思维循环”。 对失败进行反思和总结，有助于智能体在后续尝试中纠正错误并跳出这类循环。
Self-refine utilizes an iterative process of generation, feedback, and refinement. After each generation, LLM generates feedback for the plan, facilitating adjustments based on the feedback. Reflexion extends ReAct by incorporating an evaluator to assess trajectories. LLM generates self-reflections upon error detection, aiding in error correction.
Self-refine 使用生成、反馈与完善的迭代过程。 每次生成后，LLM 都会为计划生成反馈，从而根据反馈进行调整。 Reflexion 通过引入评估器来评价轨迹，从而扩展了 ReAct。 LLM 在检测到错误时生成自我反思，以帮助纠正错误。
CRITIC uses external tools like Knowledge Bases and Search Engines to validate LLM-generated actions. It then leverages external knowledge for self-correction, significantly reducing factual errors. InteRecAgent employs a mechanism called ReChain for self-correction. An LLM is used to evaluate the response and tool-using plan generated by the interactive recommendation agent, summarize feedback on errors, and decide whether to restart planning.
CRITIC 使用知识库和搜索引擎等外部工具验证 LLM 生成的动作。 随后，它利用外部知识进行自我纠正，显著减少事实错误。 InteRecAgent 使用一种名为 ReChain 的机制进行自我纠正。 它使用一个 LLM 评估交互式推荐智能体生成的响应和工具使用计划，总结错误反馈，并决定是否重新开始规划。
LEMA gathers mistaken planning samples first and employs more powerful GPT-4 for correction. Those corrected samples are then used to fine-tune the LLM-Agent, resulting in significant performance improvements across various scales of the LLaMA model.
LEMA 首先收集错误的规划样本，并使用能力更强的 GPT-4 进行纠正。 随后，这些纠正后的样本被用于微调 LLM 智能体，使不同规模的 LLaMA 模型都获得显著的性能提升。
Particularly, the self-reflective strategy bears resemblance to the principles of reinforcement learning, where the agent plays the role of the decision-maker, such as the policy network. Environmental feedback triggers updates of the policy network. However, in contrast to deep reinforcement learning where updates are achieved by modifying model parameters, in the LLM agent, this update occurs through self-reflection by the LLM itself, culminating in textual verbal feedbacks. These textual feedbacks can serve as both long-term and short-term memory, influencing the agent's subsequent planning outputs through the prompts. Nevertheless, the convergence of this textual form of update currently lacks a guaranteed proof, indicating the inability to demonstrate that continual reflection can ultimately lead the LLM agent to a specified goal.
特别是，自我反思策略与强化学习原则相似，其中智能体扮演决策者的角色，例如策略网络。 环境反馈会触发策略网络的更新。 然而，与通过修改模型参数完成更新的深度强化学习不同，在 LLM 智能体中，这种更新由 LLM 自身通过自我反思完成，最终形成文本形式的语言反馈。 这些文本反馈既可以作为长期记忆，也可以作为短期记忆，并通过提示影响智能体后续的规划输出。 尽管如此，目前还没有证明确保这种文本更新形式能够收敛，因此无法证明持续反思最终能使 LLM 智能体达到指定目标。
7. Memory-Augumented Planning
For agents, memory is a crucial pathway to enhance planning capabilities and the potential for growth. Regarding the memory mechanisms in LLM-Agents, there are currently two major approaches to enhance planning abilities through memory: RAG-based memory and embodied memory.
对于智能体而言，记忆是增强规划能力与成长潜力的重要途径。 就 LLM 智能体的记忆机制而言，目前主要有两种通过记忆增强规划能力的方法：基于 RAG 的记忆和具身记忆。
7.1 RAG-based Memory
Retrieval Augmented Generation (RAG) techniques are proposed to aid text generation with retrieved information. It is capable of enhancing the LLM with the latest knowledge, such as New Bing and Google Bard. For LLM agents, past experiences could be stored in the memory and retrieved when needed. The core idea of such methods is to retrieve task-relevant experiences from the memory during task planning. Among those methods, memories are typically stored in additional storage, and the forms are diverse, such as texts, tabular forms, knowledge graph, etc.
检索增强生成（RAG）技术通过检索到的信息辅助文本生成。 它能够用最新知识增强 LLM，例如 New Bing 和 Google Bard。 对于 LLM 智能体，过去的经验可以存储在记忆中，并在需要时检索。 这类方法的核心思想是在任务规划期间从记忆中检索与任务相关的经验。 在这些方法中，记忆通常保存在额外存储中，形式多种多样，例如文本、表格和知识图谱等。
Generative Agents store the daily experiences of human-like agents in text form and retrieve memories based on a composite score of recency and relevance to the current situation. Similarly, MemoryBank, TiM, and RecMind encode each memory using a text encoding model into a vector and establish an indexing structure, such as FAISS library. During retrieval, the description of the current status is used as a query to retrieve memories from the memory pool. The difference between the three lies in the way memories are updated.
Generative Agents 以文本形式存储类人智能体的日常经验，并根据近期性及其与当前情境相关性的综合得分检索记忆。 类似地，MemoryBank、TiM 和 RecMind 使用文本编码模型将每条记忆编码为向量，并建立诸如 FAISS 库这样的索引结构。 检索时，当前状态的描述被用作查询，从记忆池中检索记忆。 这三者的差异在于更新记忆的方式。
MemGPT leverages the concept of multiple levels of storage in computer architecture, abstracting the context of LLM into RAM and treating the additional storage structure as a disk. LLM can spontaneously decide whether to retrieve historical memories or save the current context to storage. REMEMBER stores historical memories in the form of a Q-value table, where each record is (environment, task, action, Q-value)-tuple. During retrieval, positive and negative memories are both retrieved for LLM to generate plan based on the similarity of the environment and task.
MemGPT 借鉴计算机体系结构中的多级存储概念，将 LLM 的上下文抽象为内存，并把额外的存储结构视为磁盘。 LLM 可以自主决定是检索历史记忆，还是将当前上下文保存到存储中。 REMEMBER 以 Q 值表的形式存储历史记忆，其中每条记录都是一个（环境、任务、动作、Q 值）元组。 检索时，系统会同时检索正面和负面记忆，供 LLM 根据环境和任务的相似性生成计划。
7.2 Embodied Memory
Embodied memory involves finetuning the LLM with the agent's historical experiential samples, embedding memories into the model parameters. Usually the experiential samples are collected from the agents's interactions with environment, which may consist of commonsense knowledge about the environment, task-related priors, and successful or failed experiences. While the cost of training a language model with more than billions of parameters is huge, parameter-efficient fine-tuning (PEFT) techniques are leveraged to reduce cost and speed up by training a small part of parameters only, such as LoRA, QLoRA, P-tuning, et al.
具身记忆使用智能体的历史经验样本微调 LLM，将记忆嵌入模型参数中。 经验样本通常收集自智能体与环境的交互，其中可能包括关于环境的常识知识、与任务相关的先验，以及成功或失败的经验。 虽然训练参数量超过十亿的语言模型成本高昂，但可以利用参数高效微调（PEFT）技术，只训练一小部分参数来降低成本并加快速度，例如 LoRA、QLoRA 和 P-tuning 等。
CALM utilizes ground-truth action trajectories collected from the text-world environment to finetune GPT-2 using next token prediction task, enabling it to memorize planning-related information and generalize well on planning tasks. Similarly, TDT uses collected Markov decision process data to fine-tune Text Decision Transformer (TDT). It achieves better success rates on more challenging ScienceWorld tasks. AgentTuning organizes plan trajectories from various tasks into a dialogue form to finetune the LLaMA model, showing significant improvements in performance on unseen planning tasks.
CALM 利用从文本世界环境中收集的真实动作轨迹，通过下一个 token 预测任务微调 GPT-2，使其能够记住与规划相关的信息，并在规划任务上良好泛化。 类似地，TDT 使用收集到的马尔可夫决策过程数据微调 Text Decision Transformer（TDT）。 它在更具挑战性的 ScienceWorld 任务上取得了更高的成功率。 AgentTuning 将不同任务的规划轨迹组织为对话形式来微调 LLaMA 模型，在未见过的规划任务上表现出显著的性能提升。
7.3 Discussions
The RAG-based and Fine-tuning-based memory approaches enhance LLM-Agent planning capabilities, each with distinct advantages and limitations. RAG-based methods offer real-time, low-cost external memory updates mainly in natural language text, but rely on the accuracy of retrieval algorithm. Finetuning provides a larger memorization capacity through parameter modifications but has high memory update costs and struggles with retaining fine-grained details.
基于 RAG 和基于微调的记忆方法都能增强 LLM 智能体的规划能力，但各自具有不同的优势与局限。 基于 RAG 的方法能够主要以自然语言文本的形式实时、低成本地更新外部记忆，但依赖检索算法的准确性。 微调通过修改参数提供更大的记忆容量，但记忆更新成本较高，而且难以保留细粒度细节。
Memory-enhanced LLM-Agents demonstrate enhanced growth and fault tolerance in planning, yet memory generation heavily depends on LLM's generation capabilities. Improving weaker LLM-Agents through self-generated memory remains a challenging area to explore.
记忆增强的 LLM 智能体在规划中展现出更强的成长能力和容错能力，但记忆生成高度依赖 LLM 的生成能力。 通过自生成记忆改进能力较弱的 LLM 智能体，仍是一个具有挑战性的探索方向。
8. Evaluation
| AlfWorld | ScienceWorld | HotPotQA | FEVER | |||||
|---|---|---|---|---|---|---|---|---|
| Metrics | SR (%) | EX ($) | AR | EX ($) | SR (%) | EX ($) | SR (%) | EX ($) |
| Z-CoT | N/A | N/A | N/A | N/A | 0.01 | 0.95 | 0.39 | 1.07 |
| F-CoT | 0.43 | 98.60 | 16.58 | 272.22 | 0.32 | 5.73 | 0.61 | 2.25 |
| CoT-SC | 0.57 | 105.37 | 15.24 | 274.33 | 0.33 | 7.86 | 0.62 | 3.21 |
| SayCan | 0.60 | 113.61 | 12.36 | 125.71 | N/A | N/A | N/A | N/A |
| ReAct | 0.57 | 152.18 | 15.05 | 356.03 | 0.34 | 66.00 | 0.63 | 22.20 |
| Reflexion | 0.71 | 220.17 | 19.39 | 724.48 | 0.39 | 112.49 | 0.68 | 37.26 |
Evaluating the planning capability of the agent is a critical issue in the research area. Here we investigate several mainstream benchmarking methods, categorizing them into the following types.
评估智能体的规划能力是这一研究领域中的关键问题。 这里，我们考察了几种主流基准测试方法，并将其分为以下类型。
Interactive Gaming Environments: Game environments may provide real-time multi-modal feedback based on the agent's actions, including textual and visual feedback. Currently, the most widely used gaming environment is Minecraft, where the agent needs to gather materials to create tools for obtaining more rewards. The quantity of tools created by the agent is often used as an evaluation metric. Another popular category is the text-based interactive environments, such as ALFWorld, ScienceWorld, et al, where the agent locates in an environment described in natural language, with limited actions and locations. The success rate or the rewards obtained are commonly used as evaluation metrics. Compared with Minecraft, these text-based interactive environments are often simpler, with straightforward feedback and fewer feasible actions.
交互式游戏环境： 游戏环境可以根据智能体的动作提供实时多模态反馈，包括文本反馈和视觉反馈。 目前，使用最广泛的游戏环境是 Minecraft，智能体需要在其中收集材料、制作工具，以获得更多奖励。 智能体创建的工具数量通常被用作评估指标。 另一类常见环境是 ALFWorld、ScienceWorld 等基于文本的交互式环境；智能体位于一个以自然语言描述、动作与位置有限的环境中。 成功率或获得的奖励通常被用作评估指标。 与 Minecraft 相比，这些基于文本的交互式环境通常更简单，反馈直接，可行动作也更少。
Interactive Retrieval Environments: Interactive retrieval environments simulate the process of information retrieval and reasoning that humans undergo in real life. In these environments, agents are often allowed to interact with search engines and other web services, using actions such as searching keywords or executing click, forward, and backward operations to acquire more information, thereby obtaining answers to questions or completing information retrieval tasks. Commonly used retrieval environments include question-answering tasks based on the Wikipedia engine (such as HotPotQA and Fever) and web browsing tasks to find specific information, including WebShop, Mind2Web, and WebArena. The task success rate is usually used as the metric.
交互式检索环境： 交互式检索环境模拟人类在现实生活中进行信息检索与推理的过程。 在这些环境中，智能体通常可以与搜索引擎和其他网络服务交互，通过搜索关键词或执行点击、前进和后退等操作来获取更多信息，从而回答问题或完成信息检索任务。 常用的检索环境包括基于 Wikipedia 引擎的问答任务（如 HotPotQA 和 Fever），以及寻找特定信息的网页浏览任务，包括 WebShop、Mind2Web 和 WebArena。 任务成功率通常被用作评估指标。
Interactive Programming Environments: Interactive programming environments simulate the interaction between programmers and computers, testing the agent's planning ability in solving computer-related problems. In these environments, agents are required to interact with computers to solve problems by writing code or instructions. They would receive various feedback including compile and runtime error messages, as well as execution results. Popular interactive programming environments involve issues related to operating systems, databases, etc., such as Agent Bench, MiniWoB++.
交互式编程环境： 交互式编程环境模拟程序员与计算机之间的交互，用于测试智能体解决计算机相关问题时的规划能力。 在这些环境中，智能体需要通过编写代码或指令与计算机交互来解决问题。 它们会收到多种反馈，包括编译错误、运行时错误消息以及执行结果。 常见的交互式编程环境涉及操作系统、数据库等问题，例如 Agent Bench 和 MiniWoB++。
Most of these existing interactive environments lack fine-grained evaluation, where the performance is predominantly evaluated by the final success rate. Furthermore, unlike real-world scenarios where there are often multiple paths to complete a task, there is typically only one "golden" path in most simulated environments due to the high annotation cost.
现有交互式环境大多缺乏细粒度评估，性能主要通过最终成功率来评价。 此外，与现实世界中通常存在多条任务完成路径不同，由于标注成本高昂，大多数模拟环境通常只有一条“标准”路径。
Experiments. We have conducted experiments on four benchmarks to validate the performance of representative works, shown in Table 2. We have implemented six prompt-based methods due to limited budgets, covering task decomposition, multi-path selection, and reflection. As for the benchmarks, ALFWorld, ScienceWorld, HotPotQA, and FEVER are employed, involving interactive gaming and question-answering benchmarks.
实验。 我们在四个基准上开展实验，以验证代表性工作的性能，结果见 表 2。 由于预算有限，我们实现了六种基于提示的方法，涵盖任务分解、多路径选择和反思。 基准方面，我们采用 ALFWorld、ScienceWorld、HotPotQA 和 FEVER，涵盖交互式游戏与问答基准。
Since ALFWorld and ScienceWorld are involved in larger action space, the zero-shot method, i.e. ZeroShot-CoT, is not applicable due to unawareness of action space. SayCan improves CoT by grounding output actions into action space with a value function, which does not apply to QA tasks because there are only two actions: Search[keyword] and Lookup[keyword]. And we set the value function as a textual embedding model bge-small-en-v1.5. We obtain 3 actions and 5 answers each step for gaming tasks and QA tasks for CoT-SC, respectively. The round of retries in Reflexion is set to 1. We use the API of text-davinci-003 in OpenAI as LLM.
由于 ALFWorld 和 ScienceWorld 涉及更大的动作空间，零样本方法 ZeroShot-CoT 因不了解动作空间而不适用。 SayCan 使用价值函数将输出动作约束到动作空间中，从而改进 CoT；但这不适用于问答任务，因为问答任务只有两个动作：Search[keyword] 和 Lookup[keyword]。 我们将价值函数设为文本嵌入模型 bge-small-en-v1.5。 对于 CoT-SC，我们在游戏任务和问答任务的每一步分别获取 3 个动作和 5 个答案。 Reflexion 的重试轮数设为 1。 我们使用 OpenAI 的 text-davinci-003 API 作为 LLM。
(i) The performance increases with the expenses. As CoT-SC, ReAct and Reflexion are involved in multiple plans, additional thoughts, and reflections, respectively, their expenses are more than their backbone methods. Intuitively, more tokens represent more detailed thinking, resulting in performance improvements.
（i）性能随费用增加而提高。 CoT-SC、ReAct 和 Reflexion 分别涉及多个计划、额外思考和反思，因此它们的费用高于各自的基础方法。 直观而言，更多 token 意味着更细致的思考，从而带来性能提升。
(ii) Fewshot examples are suggested for complicated tasks. Despite that the magic instruction Let's think step by step can lead to more reasoning, ZeroShot-CoT exhibits severe performance degradation in two QA benchmarks, which demonstrates the necessity of the examples for LLM to further understand the task.
（ii）复杂任务建议使用少样本示例。 尽管魔法指令 Let's think step by step 可以引导更多推理，但 ZeroShot-CoT 在两个问答基准上的性能严重下降，这说明示例对于 LLM 进一步理解任务是必要的。
(iii) Reflection plays a crucial role in improving the success rate, especially for complex tasks. Despite Reflexion consuming about twice the tokens compared with ReAct, the improvements in complicated tasks are promising, such as ALFWorld and ScienceWorld, which shows that LLM possesses the error-correcting capability.
（iii）反思对于提高成功率至关重要，尤其是在复杂任务中。 尽管 Reflexion 消耗的 token 约为 ReAct 的两倍，但它在 ALFWorld 和 ScienceWorld 等复杂任务上的提升很有前景，这表明 LLM 具备纠错能力。
9. Conclusions and Future Directions
Since LLM has shown the emergence of intelligence, there has been an increasing focus on using LLM to enhance the planning capabilities of agents. The major directions are summarized in Figure 1, with a detailed comparison and analysis of various methods presented in Sections 3 to 7. We also conducted experiments on four benchmarks, comparing the effectiveness of several representative methods and showing that performance increases with expenses. Despite the enhancements made by these works in planning capabilities, there are still some significant challenges.
随着 LLM 展现出智能涌现，人们越来越关注使用 LLM 增强智能体的规划能力。 主要方向总结于 图 1，第 3 至第 7 节对各种方法进行了详细比较与分析。 我们还在四个基准上开展实验，比较若干代表性方法的有效性，并表明性能随费用增加而提高。 尽管这些工作增强了规划能力，但仍然存在一些重大挑战。
Hallucinations. During the planning process, LLM often suffers from hallucinations, leading to irrational plans, unfaithfulness to task prompts, or failing to follow complex instructions. For instance, plans may include actions that interact with items not existed in the environment. Although these issues can be alleviated through careful prompt engineering, they reflect fundamental shortcomings in LLM.
幻觉。 在规划过程中，LLM 经常受到幻觉影响，导致计划不合理、不忠实于任务提示，或无法遵循复杂指令。 例如，计划可能包含与环境中不存在的物品交互的动作。 虽然这些问题可以通过谨慎的提示工程得到缓解，但它们反映了 LLM 的根本缺陷。
Feasibility of Generated Plans. LLM, being fundamentally based on statistical learning, optimizes the probability of the next word through massive data. Compared to symbolic artificial intelligence, this approach struggles to obey complex constraints, especially when dealing with less common constraints encountered during LLM training. Consequently, plans generated by LLM may lack feasibility without considering adequate preconditions. Connecting LLM with symbolic planning models without altering LLM itself is a promising future direction.
生成计划的可行性。 LLM 从根本上基于统计学习，通过海量数据优化下一个词的概率。 与符号人工智能相比，这种方法难以遵循复杂约束，尤其是在处理 LLM 训练期间较少遇到的约束时。 因此，如果没有考虑充分的前置条件，LLM 生成的计划可能缺乏可行性。 在不改变 LLM 本身的情况下，将 LLM 与符号规划模型连接起来，是一个很有前景的未来方向。
Efficiency of Generated Plans. Generating efficient plans is a crucial issue in planning. However, in existing LLM agents, planning is greedily based on generated plans from LLM output, without considering the efficiency of the generated plans. Therefore, future developments may require introducing additional efficiency evaluation modules to work in conjunction with LLM for more efficient plans.
生成计划的效率。 生成高效计划是规划中的一个关键问题。 然而，在现有 LLM 智能体中，规划贪心地依赖 LLM 输出的生成计划，而没有考虑这些计划的效率。 因此，未来发展可能需要引入额外的效率评估模块，与 LLM 协同工作，以生成更高效的计划。
Multi-Modal Environment Feedback. LLM is originally designed for processing textual inputs, but real-world environment feedback is often multi-modal, including images, audio, etc., which are challenging to describe in natural language. Therefore, LLM agents face limitations when handling such scenarios. Future considerations may involve integrating the development of multi-modal large models and revisiting related planning strategies.
多模态环境反馈。 LLM 最初为处理文本输入而设计，但现实世界的环境反馈通常是多模态的，包括图像、音频等，而这些内容很难用自然语言描述。 因此，LLM 智能体在处理此类场景时存在局限。 未来可以结合多模态大模型的发展，重新审视相关规划策略。
Fine-grained Evaluation. As mentioned in Section 8, existing benchmarks mostly rely on the final completion status of tasks, lacking fine-grained step-wise evaluations. Additionally, environmental feedback is often rule-based, simplistic, and distant from real-world scenarios. A potential future direction is to leverage high-intelligence models like LLM to design more realistic evaluation environments.
细粒度评估。 如第 8 节所述，现有基准主要依赖任务的最终完成状态，缺乏细粒度的逐步评估。 此外，环境反馈往往基于规则、过于简单，并且与现实世界场景相距甚远。 一个潜在的未来方向，是利用 LLM 等高智能模型设计更真实的评估环境。