From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models
MemoryFine-Tuning贝壳找房从 LLM 到对话智能体：结合大语言模型微调的记忆增强架构
Abstract
This paper introduces RAISE (Reasoning and Acting through Scratchpad and Examples), an advanced architecture enhancing the integration of Large Language Models (LLMs) like GPT-4 into conversational agents. RAISE, an enhancement of the ReAct framework, incorporates a dual-component memory system, mirroring human short-term and long-term memory, to maintain context and continuity in conversations. It entails a comprehensive agent construction scenario, including phases like Conversation Selection, Scene Extraction, CoT Completion, and Scene Augmentation, leading to the LLMs Training phase. This approach appears to enhance agent controllability and adaptability in complex, multi-turn dialogues. Our preliminary evaluations in a real estate sales context suggest that RAISE has some advantages over traditional agents, indicating its potential for broader applications. This work contributes to the AI field by providing a robust framework for developing more context-aware and versatile conversational agents.
本文提出 RAISE（Reasoning and Acting through Scratchpad and Examples，通过草稿区与示例进行推理和行动），这是一种先进架构，用于加强 GPT-4 等大语言模型（LLM）与对话智能体的结合。 作为 ReAct 框架的增强版本，RAISE 引入了一个模仿人类短期记忆和长期记忆的双组件记忆系统，以维持对话中的上下文与连续性。 它包含一套完整的智能体构建流程，其中包括对话筛选、场景提取、思维链补全和场景增强等阶段，并最终进入 LLM 训练阶段。 这种方法似乎能够提升智能体在复杂多轮对话中的可控性和适应性。 我们在房地产销售场景中的初步评估表明，RAISE 相比传统智能体具有一定优势，显示出应用于更广泛领域的潜力。 这项工作为构建上下文感知能力更强、用途更灵活的对话智能体提供了一个稳健框架，从而为人工智能领域作出贡献。
1. Introduction
The landscape of Artificial Intelligence (AI) is continuously evolving, with Large Language Models (LLMs) emerging as pivotal components in the advancement towards Artificial General Intelligence (AGI). These models, exemplified by GPT-4 and similar architectures, have demonstrated remarkable proficiency in a range of tasks, from conversation and reasoning to complex problem-solving in various domains. The versatility of LLMs has been further enriched by innovative prompting strategies and the integration of external tools, enhancing their capabilities beyond basic language processing.
人工智能（AI）领域在持续演进，大语言模型（LLM）正成为迈向通用人工智能（AGI）过程中的关键组成部分。 以 GPT-4 及类似架构为代表的这些模型，在从对话和推理到不同领域复杂问题求解的一系列任务上展现了卓越能力。 创新提示策略和外部工具的结合进一步丰富了 LLM 的通用性，使其能力超越了基本语言处理。
However, a significant challenge in the realm of LLMs lies in their integration into conversational agents. While these models exhibit high levels of performance in isolated tasks, creating an agent that can sustain coherent, context-aware, and purpose-driven conversations remains an intricate endeavor. The need for a more sophisticated framework that leverages the strengths of LLMs while addressing their limitations in conversational settings has become increasingly apparent.
然而，LLM 领域的一项重大挑战在于如何将它们集成到对话智能体中。 尽管这些模型在孤立任务中表现出色，但要创建一个能够持续进行连贯、上下文感知且目标导向对话的智能体，仍是一项复杂工作。 人们越来越清楚地意识到，需要一种更完善的框架，在利用 LLM 优势的同时解决其在对话场景中的局限。
In response to this need, we introduce the RAISE (Reasoning and Acting through Scratchpad and Examples) architecture. RAISE represents a refined enhancement of the existing ReAct framework, specifically designed to augment the capabilities of conversational agents. This paper presents a detailed exploration of RAISE, highlighting its unique components and the benefits it offers in the development of conversational agents.
为满足这一需求，我们提出 RAISE（Reasoning and Acting through Scratchpad and Examples）架构。 RAISE 是对现有 ReAct 框架的进一步增强，专门用于提升对话智能体的能力。 本文详细探讨 RAISE，重点介绍其独特组件及其为对话智能体开发带来的益处。
The cornerstone of RAISE is its incorporation of a dual-component memory system, analogous to the human brain's short-term and long-term memory functions. The Scratchpad component functions as a transient storage, capturing and processing key information and conclusions from recent interactions, akin to short-term memory. In parallel, the retrieval module operates as the agent's long-term memory, sourcing and incorporating examples relevant to the current conversational context. This enhanced memory mechanism can flexibly bolster the capabilities of conversational AI, and also provides a convenient interface for humans to customize and control the behavior of conversational AI systems.
RAISE 的基石是引入了一个双组件记忆系统，该系统类似于人脑的短期记忆和长期记忆功能。 Scratchpad 组件充当临时存储，捕获并处理近期交互中的关键信息和结论，其作用类似于短期记忆。 与此同时，检索模块作为智能体的长期记忆，获取并引入与当前对话上下文相关的示例。 这种增强记忆机制可以灵活提升对话 AI 的能力，也为人类定制和控制对话 AI 系统的行为提供了便捷接口。
Furthermore, the RAISE architecture is founded on a comprehensive agent construction scenario, emphasizing the creation of conversational agents from scratch to ensure authenticity and relevance in real-world interactions. This paper delineates the RAISE methodology, encompassing a sequence of meticulously orchestrated phases. These include Conversation Selection, Scene Extraction, CoT (Chain of Thought) Completion, and Scene Augmentation, all leading up to the pivotal LLMs Training phase. This structured approach is instrumental in developing agents that excel not only in language processing but also in contextual awareness and adaptability, catering to a spectrum of conversational dynamics.
此外，RAISE 架构建立在一套完整的智能体构建流程之上，强调从零开始创建对话智能体，以确保真实世界交互中的真实性与相关性。 本文阐述了 RAISE 方法，其中包含一系列经过精心组织的阶段。 这些阶段包括对话筛选、场景提取、思维链（CoT）补全和场景增强，并最终进入关键的 LLM 训练阶段。 这种结构化方法有助于开发不仅擅长语言处理，而且具有上下文感知能力和适应性的智能体，从而应对各种对话动态。
Our experimental evaluations, conducted on a specialized in-house dataset focused on real estate sales, demonstrate the superiority of RAISE over conventional conversational agents. The results showcase RAISE's ability to handle complex, multi-turn conversations with enhanced context awareness and adaptability. While our experiments are centered on the real estate domain, the principles and methodologies underpinning RAISE are universally applicable, making it a versatile framework for various applications.
我们在一个专注于房地产销售的内部专用数据集上进行了实验评估，结果证明 RAISE 优于传统对话智能体。 结果展示了 RAISE 处理复杂多轮对话的能力，并体现出更强的上下文感知能力和适应性。 尽管我们的实验聚焦于房地产领域，但支撑 RAISE 的原则和方法具有普遍适用性，使其成为可用于多种应用的灵活框架。
In summary, this paper presents the following contributions:
- We introduce RAISE, a refined enhancement of the ReAct framework, which utilizes scratchpad and retrieved examples to augment the agent's capabilities.
- We propose a fine-tuning scenario for Large Language Models (LLMs) within RAISE, which, compared to the use of prompts alone, not only enhances the controllability of the agent but also improves its effectiveness and efficiency.
- Through experiments conducted on our in-house dataset, we demonstrate RAISE's superiority as a conversational agent. While our experiments are concentrated on real estate sales, the underlying principles and methodologies of RAISE have wide-ranging applications and can be adapted to various domains, highlighting its versatility.
总而言之，本文作出以下贡献：
- 我们提出 RAISE，这是对 ReAct 框架的进一步增强，它利用草稿区和检索到的示例来提升智能体能力。
- 我们提出 RAISE 中的 LLM 微调流程；与仅使用提示相比，该流程不仅增强了智能体的可控性，还提升了其有效性和效率。
- 通过在内部数据集上开展实验，我们证明了 RAISE 作为对话智能体的优越性。尽管实验聚焦于房地产销售，但 RAISE 的底层原则和方法具有广泛用途，可以适配不同领域，体现了其通用性。
2. Agent Framework
Inspired by ReAct, we introduce RAISE architecture, as shown in Figure 1. The architecture primarily encompasses the following components.
受 ReAct 启发，我们提出 RAISE 架构，如图1所示。 该架构主要包含以下组件。
2.1 Dialogue
The dialogue module serves as the core interface for user-agent communication. It handles incoming user queries and delivers tailored responses formulated by the agent.
对话模块是用户与智能体沟通的核心接口。 它处理传入的用户查询，并交付由智能体生成的定制响应。
2.2 LLMs
As the agent's brain, the LLMs requires capabilities for perception, task-specific planning, tool usage, and summarization. These skills can be developed on the LLMs using either prompt engineering or fine-tuning methods. Our study has conducted comparative experiments to stimulate these capabilities, utilizing models such as GPT-4, GPT-3.5, and Qwen-14B-Chat. This paper explores the strengths and limitations of each model in handling specific task types and provides concrete metrics for evaluating their performance.
作为智能体的大脑，LLM 需要具备感知、任务特定规划、工具使用和总结能力。 这些技能可以通过提示工程或微调方法在 LLM 上培养。 我们的研究使用 GPT-4、GPT-3.5 和 Qwen-14B-Chat 等模型开展比较实验，以激发这些能力。 本文探讨每个模型在处理特定任务类型时的优势和局限，并提供评估其性能的具体指标。
2.3 Memory
The memory module in RAISE framework stores information perceived from its environment and facilitates the agent's future actions. The memory includes the following components:
RAISE 框架中的记忆模块存储从环境中感知的信息，并为智能体未来的行动提供支持。 记忆包含以下组件：
System Prompt. Includes profiles (detailing role identity, objectives, and behaviors), task instructions, tool descriptions, and few-shot learning elements for optimizing model performance. Flexibly designed, system prompt can either remain static or dynamically adjust to accommodate various stages of a dialogue and differing query types.
系统提示词。 包含画像（详细描述角色身份、目标和行为）、任务指令、工具描述以及用于优化模型性能的少样本学习元素。 系统提示词采用灵活设计，可以保持静态，也可以动态调整，以适应对话的不同阶段和不同查询类型。
Context. Includes conversation history and task trajectory. Conversation history records all query-response pairs within the dialogue, providing a complete context for more accurate agent perception. Task trajectory documents the decision-making trajectory, including plan designation, tool selection, and execution, guiding the agent's future planning.
上下文。 包含对话历史和任务轨迹。 对话历史记录对话中的所有查询—响应对，为智能体进行更准确的感知提供完整上下文。 任务轨迹记录决策轨迹，包括计划制定、工具选择和执行，从而指导智能体未来的规划。
Scratchpad. Logs background information, knowledge generated by reasoning and observations from previous tool usage, essential for efficiency in multi-turn interactions.
草稿区。 记录背景信息、推理产生的知识以及先前工具使用的观察结果，这对于提升多轮交互效率至关重要。
Examples. Comprises query-response pairs used for recalling relevant examples to supplement the model's and tools' knowledge gaps and to customize agent behavior and expression.
示例。 由查询—响应对组成，用于召回相关示例，以补充模型和工具的知识缺口，并定制智能体的行为和表达方式。
These four components collectively form the working memory of RAISE, with conversation history and scratchpad being dialogue-level, while examples and task trajectory are turn-level.
这四个组件共同构成 RAISE 的工作记忆，其中对话历史和草稿区位于对话级，而示例和任务轨迹位于轮次级。
2.4 Tool
The tool module enriches LLMs after pretraining and Supervised Fine-Tuning (SFT) by integrating external knowledge sources and resources. This module incorporates a diverse array of tools, including but not limited to databases for data retrieval, APIs for system interactions, sophisticated recommendation systems, and collaborative frameworks involving other LLMs or agents. The description file for a tool typically needs to include the tool's name, its function, essential parameters, optional parameters, and may also include some usage examples. This descriptive file aids agents in better planning, tool selection, parameter generation for tools, and execution of those tools.
工具模块通过结合外部知识源和资源，丰富经过预训练和监督微调（SFT）后的 LLM。 该模块包含多种工具，包括但不限于用于数据检索的数据库、用于系统交互的 API、复杂的推荐系统，以及涉及其他 LLM 或智能体的协作框架。 工具描述文件通常需要包含工具名称、功能、必要参数和可选参数，也可以包含一些使用示例。 该描述文件帮助智能体更好地进行规划、工具选择、工具参数生成和工具执行。
2.5 Controller: Control Agent Loop
The controller module connects the aforementioned modules through preset trigger conditions. Upon receiving a new query, the agent executes the loop of perception, planning, tool selection, and tool execution. The specific process is as follows.
控制器模块通过预设触发条件连接上述模块。 收到新查询后，智能体执行感知、规划、工具选择和工具执行循环。 具体过程如下。
Memory Update. At the beginning of a conversation, the Scratchpad records the context of the dialogue, including user and agent roles, date, time, etc. During the conversation, each time a user query is received, the system will: (1) Add the user's query to the Conversation History; (2) Recall top-
记忆更新。 在对话开始时，草稿区记录对话上下文，包括用户和智能体角色、日期、时间等。 在对话期间，每次收到用户查询时，系统都会：(1) 将用户查询添加到对话历史；(2) 基于历史查询和当前查询，使用向量检索从示例池中召回当前任务的 top-
Task Planning. After collecting the above information, it is combined into a complete task inference prompt according to the designed template, as illustrated in Figure 2. An example of the complete prompt is available in the Appendix. The LLM utilizes the information within the prompt for perception and planning, subsequently outputting actions in accordance with the format outlined in the prompt. If an action involves invoking a tool, it should specify the tool's name and input parameters.
任务规划。 收集上述信息后，系统按照设计的模板将其组合成完整的任务推理提示，如图2所示。 完整提示示例见附录。 LLM 利用提示中的信息进行感知和规划，随后按照提示规定的格式输出动作。 如果某个动作涉及调用工具，则应指定工具名称和输入参数。
Tool Execution. This phase involves executing the tool selected in the previous step. The command for tool execution may either be directly output by the agent or correspond to a manually crafted function specific to each tool. The output of the execution is formatted as predetermined.
工具执行。 该阶段执行上一步选择的工具。 工具执行命令可以由智能体直接输出，也可以对应于为每个工具手动编写的专用函数。 执行输出按照预定格式进行组织。
Summary. The agent, synthesizing all the information gathered from the environment, decides whether it can respond to the user's query. Termination criteria might include having gathered sufficient information, exceeding a preset number of loops, or encountering a system error. Upon meeting any of these conditions, the agent can proceed to summarize its findings and provide a response.
总结。 智能体综合从环境中收集的所有信息，判断自己能否响应用户查询。 终止条件可能包括已经收集到足够信息、超过预设循环次数或遇到系统错误。 满足任一条件后，智能体即可总结其发现并提供响应。
3. Agent Tuning
Section 2 presented the RAISE architecture, establishing a hardware base for agents in complex dialogues. This section shifts focus to software enhancements for RAISE, particularly activating LLMs as the agent's core. Despite the success of open-source LLMs in various tasks, studies reveal their limitations in real-world scenarios, especially compared to GPT-3.5 and GPT-4. Addressing this gap, this paper introduces a versatile finetuning method suitable for complex agent applications.
第 2 节介绍了 RAISE 架构，为复杂对话中的智能体建立了硬件基础。 本节转而关注 RAISE 的软件增强，尤其是激活作为智能体核心的 LLM。 尽管开源 LLM 在多种任务上取得成功，但研究揭示了它们在真实世界场景中的局限，尤其是与 GPT-3.5 和 GPT-4 相比。 为弥合这一差距，本文提出一种适用于复杂智能体应用的通用微调方法。
3.1 Build Datasets
The creation of training data entails significant costs. Our objective is to finetune the model efficiently using a compact yet high-quality dataset that precisely aligns with specific role-based behavioral logic. The dataset must fulfill these criteria:
创建训练数据需要付出高昂成本。 我们的目标是使用一个紧凑但高质量、能够精确对齐特定角色行为逻辑的数据集来高效微调模型。 该数据集必须满足以下标准：
Authenticity. It should closely mimic real-life scenarios. Diversity. The data should encompass a wide range of scenarios. High Quality. The data must have an accurate Chain of Thought (CoT) process, encompassing aspects like planning, tool utilization, and response formulation.
真实性。 数据应当紧密模拟真实生活场景。 多样性。 数据应覆盖广泛的场景。 高质量。 数据必须包含准确的思维链（CoT）过程，涵盖规划、工具使用和响应生成等方面。
As shown in Figure 3, our proposed pipeline comprises several stages, including Conversation Selection, Scene Extraction, CoT Completion, and Scene Augmentation. The details of each stage are as follows:
如图3所示，我们提出的流水线包含多个阶段，包括对话筛选、场景提取、思维链补全和场景增强。 各阶段的细节如下：
3.1.1 Conversation Selection
To emulate specific roles in real scenarios, we start by filtering conversations from authentic dialogues based on criteria such as scene completion, a minimum number of dialogue turns, high conversation quality, and a threshold for user message ratio. These selected dialogues are then anonymized for further processing, as shown in Figure 3(a).
为了模拟真实场景中的特定角色，我们首先根据场景完整度、最少对话轮数、较高对话质量和用户消息比例阈值等标准，从真实对话中筛选会话。 随后对选中的对话进行匿名化处理，以便进一步加工，如图3(a)所示。
3.1.2 Scene Extraction
Each round of interaction serves as a segmentation point, dividing the previously selected dialogues into multiple samples, as shown in Figure 3(b). Each sample is an original scene (defined as
每轮交互都作为一个分割点，将此前选中的对话划分为多个样本，如图3(b)所示。 每个样本都是一个原始场景（定义为
Subsequently, to ensure diversity, we perform sampling based on dialogue turn counts and the intents behind user queries, resulting in a dataset rich in varied scene types.
随后，为确保多样性，我们根据对话轮数和用户查询背后的意图进行采样，从而得到包含丰富场景类型的数据集。
3.1.3 CoT Completion
In refining the training data for the RAISE framework, the next phase involves enhancing the original scenes with a CoT process, which bridges the gap between user queries and character responses. This CoT process encompasses perception, planning, tool selection, and execution. Studies have demonstrated GPT-4's efficacy in generating high-quality CoT prompts for intricate scenarios. In this study, we initially utilize GPT-4 for automated generation, followed by meticulous manual validation of the output. To assist GPT-4 in consistently generating CoT processes, we incorporate additional elements such as predefined profiles, tools, and few-shot examples into the original scene, which collectively shape the construction of the prompt, as shown in Figure 3(c). The refined complete scene thus includes the following elements:
在完善 RAISE 框架的训练数据时，下一阶段使用思维链过程增强原始场景，以弥合用户查询与角色响应之间的差距。 该思维链过程涵盖感知、规划、工具选择和执行。 研究已经证明 GPT-4 能够为复杂场景生成高质量的思维链提示。 在本研究中，我们首先使用 GPT-4 自动生成，再对输出进行细致的人工验证。 为了帮助 GPT-4 稳定生成思维链过程，我们在原始场景中加入预定义画像、工具和少样本示例等额外元素，这些元素共同构成提示，如图3(c)所示。 完善后的完整场景包含以下元素：
3.1.4 Scene Augmentation
While the Scene Extraction phase ensured diversity through actual data sampling and the CoT Completion phase added the necessary CoT intricacies, two critical challenges still need addressing:
尽管场景提取阶段通过真实数据采样保证了多样性，思维链补全阶段也加入了必要的思维链细节，但仍有两个关键挑战需要解决：
Role Hallucination. LLMs, endowed with vast domain knowledge from pre-training and fine-tuning, exhibit extensive capabilities. However, if left unchecked, our trained agents might retain these broad skills, which could conflict with their intended functional roles. For example, an agent designed to provide sales services might erroneously possess skills like coding in Python or offering recipe advice. To counter this, we introduce specific scenarios that teach the agent its capability limits, essentially making it 'unlearn' the general abilities of LLMs within these defined contexts.
角色幻觉。 LLM 从预训练和微调中获得了广泛的领域知识，因而具备丰富能力。 然而，如果不加约束，我们训练的智能体可能会保留这些广泛技能，而它们可能与智能体预期的功能角色相冲突。 例如，一个用于提供销售服务的智能体可能错误地具备用 Python 编程或提供食谱建议等技能。 为解决这一问题，我们引入特定场景来教会智能体认识自身能力边界，本质上使其在这些指定上下文中“遗忘”LLM 的通用能力。
Knowledge Hallucination. This phenomenon involves creating unrealistic or incorrect statements due to inadequate or misapplied knowledge acquired during pre-training. To mitigate this, we incorporate scenarios where the agent, despite tool utilization, still lacks essential factual knowledge, resulting in inability to respond accurately. In instances requiring factual accuracy, the agent should base its responses on knowledge acquired from its working memory or through tool interaction, rather than relying on its pre-trained database.
知识幻觉。 这一现象是指由于预训练获得的知识不足或使用不当而生成不现实或不正确的陈述。 为缓解这一问题，我们加入一些场景：智能体即便使用了工具，仍然缺少必要的事实知识，因而无法准确作答。 在需要事实准确性的情况下，智能体应根据从工作记忆或工具交互中获得的知识作答，而不是依赖其预训练知识库。
To overcome these issues, we perform data augmentation on these two categories of data, which are not included in the real online datasets, as demonstrated in Figure 3(d).
为解决这些问题，我们对真实线上数据集中没有包含的这两类数据进行增强，如图3(d)所示。
3.2 LLMs Training
Following the previous phase, we have acquired a dataset characterized by both high quality and diversity. Each sample in this dataset, identified as
经过上一阶段后，我们获得了一个兼具高质量和多样性的数据集。 该数据集中的每个样本都记为
These instances are then processed into a format conducive for full-parameter fine-tuning of open-source LLMs. Our experiments have led to an encouraging discovery: by constructing a modest amount (<
随后，这些实例被处理为适合对开源 LLM 进行全参数微调的格式。 我们的实验得到了一项令人鼓舞的发现：只需构建少量（<
4. Experiments
To demonstrate the effectiveness of the RAISE architecture and the fine-tuning method proposed in this paper in complex real-world scenarios, we conducted experiments in a real estate online Instant Messaging (IM) dialogue setting. In this scenario, the user is a customer inquiring about real estate purchases, and the agent assumes the role of a real estate consultant. A detailed introduction to the dataset, toolset, and the method of activating agent capabilities in LLMs is provided below.
为了证明本文提出的 RAISE 架构和微调方法在复杂真实世界场景中的有效性，我们在房地产线上即时通信（IM）对话场景中开展实验。 在该场景中，用户是咨询购房事宜的客户，智能体扮演房地产顾问。 下面详细介绍数据集、工具集以及在 LLM 中激活智能体能力的方法。
4.1 Datasets
To ascertain the effectiveness of the RAISE framework, we conducted comparative evaluations with various architectures, including Act-Only, ReAct, ReAct+Scratchpad, ReAct+Examples, and RAISE. This comparison aimed to ensure fair evaluation across different models, maintaining uniform dialogue scenarios and consistent additional knowledge in identical training samples across various datasets. The full trajectories for a single scenario under each architecture are depicted in Figure 4.
为了确定 RAISE 框架的有效性，我们对多种架构进行了比较评估，包括 Act-Only、ReAct、ReAct+Scratchpad、ReAct+Examples 和 RAISE。 该比较旨在确保不同模型之间的公平评估，在不同数据集中使用相同训练样本时，保持统一的对话场景和一致的附加知识。 图4展示了每种架构在同一场景下的完整轨迹。
Initially, we generated the ReAct architecture dataset following the procedure outlined in Section 3.1. We then modified this data to create training sets for the other architectures. The methodologies applied were as follows:
首先，我们按照第 3.1 节所述流程生成 ReAct 架构数据集。 随后，我们修改这些数据，为其他架构创建训练集。 所采用的方法如下：
Act-Only. This architecture was formed by removing the 'thought' process from ReAct, allowing for straightforward generation through coding.
Act-Only。 该架构通过从 ReAct 中移除“thought”过程形成，因此可以直接通过编程生成。
ReAct+Scratchpad. Building upon ReAct, the initial Scratchpad distribution comprised 20% empty, 30% partially informative, and 50% fully informative content. The ReAct data, when combined with varying output requirements, served as prompts for the regeneration of the CoT process using GPT-4.
ReAct+Scratchpad。 在 ReAct 基础上，初始草稿区的分布为 20% 空白、30% 包含部分信息、50% 包含完整信息。 ReAct 数据与不同输出要求组合后，作为提示供 GPT-4 重新生成思维链过程。
ReAct+Examples. Similar to ReAct+Scratchpad, with the distribution of Examples set at 20% empty, 30% partially informative, and 50% fully informative. The ReAct data, merged with diverse output requirements, were reformulated into prompts for GPT-4-driven CoT regeneration.
ReAct+Examples。 与 ReAct+Scratchpad 类似，示例的分布设为 20% 空白、30% 包含部分信息、50% 包含完整信息。 ReAct 数据与多样化输出要求合并后，被重新组织为提示，用于由 GPT-4 驱动的思维链再生成。
RAISE. This model integrated aspects of both ReAct+Scratchpad and ReAct+Examples. The CoT process was similarly regenerated using GPT-4.
RAISE。 该模型整合了 ReAct+Scratchpad 和 ReAct+Examples 的组成部分。 思维链过程同样使用 GPT-4 重新生成。
This structured approach enabled a thorough evaluation of each architectural element within the RAISE framework, clearly demonstrating the incremental benefits introduced by each component. Following the outlined procedure, a total of 948 scenes were generated. Out of these, 100 were randomly selected to serve as the evaluation set, while the remaining 848 instances were used for fine-tuning the model.
这种结构化方法能够全面评估 RAISE 框架中的每个架构元素，清楚展示各组件带来的增量收益。 按照上述流程，共生成 948 个场景。 其中随机选择 100 个作为评估集，其余 848 个实例用于微调模型。
4.2 Tools
Based on real estate online IM conversations, we have abstractly defined the following 12 tools, each including the tool name, input parameters, and functions:
基于房地产线上 IM 对话，我们抽象定义了以下 12 种工具，每种工具都包含工具名称、输入参数和功能：
- Real Estate Consultant Information
: Retrieves the consultant's name, contact details, WeChat ID, ranking, performance metrics, and more.
- House Information
: Offers essential details about a property, including its size, price, floor level, school district presence, and renovation status.
- Community Information
: Provides insights into the community, covering aspects like green spaces, property management, building specifications, proximity to subway stations, schools, and medical facilities.
- House Layout Analysis
: Analyzes the strengths and weaknesses of a property's layout.
- House Price Changes
: Tracks price fluctuations for a specific property.
- Community Price Changes
: Reports on average price trends within a particular community.
- Community Transactions
: Accesses recent transaction data from the same community.
- Tax Policy
: Updates on the latest tax regulations and implications.
- Loan Policy
: Delivers current information on loan policies.
- Market Analysis
: Provides up-to-date real estate market insights.
- Recommend Listings
: Suggests property listings to customers based on their conversation history and inferred needs, including rationale for each recommendation.
- Value Report
: Generates a comprehensive value report card for a property, aimed at engaging customers and encouraging them to share their contact details.
- 房地产顾问信息
：检索顾问姓名、联系方式、微信号、排名、业绩指标等信息。
- 房源信息
： 提供房产的基本信息，包括面积、价格、楼层、是否为学区房以及装修状态。
- 小区信息
：提供小区相关信息，包括绿化、物业管理、建筑规格，以及与地铁站、学校和医疗设施的距离等方面。
- 户型分析
：分析房产户型的优势和不足。
- 房源价格变化
：跟踪特定房产的价格波动。
- 小区价格变化
：报告特定小区的平均价格趋势。
- 小区成交记录
：访问同一小区近期的成交数据。
- 税收政策
：提供最新税收规定及其影响。
- 贷款政策
：提供当前贷款政策信息。
- 市场分析
：提供最新房地产市场洞察。
- 推荐房源
：根据客户的对话历史和推断出的需求推荐房源，并给出每项推荐的理由。
- 价值报告
：为房产生成完整价值报告卡，用于吸引客户并鼓励其分享联系方式。
4.3 LLMs
The models used in this study are as follows:
本研究使用的模型如下：
OpenAI GPT. We utilized GPT-4 for generating all fine-tuning data and employed both GPT-3.5 and GPT-4 for prompting purposes. Both models were operated in ChatCompletion mode as of November 2023, with the temperature set to 0.5.
OpenAI GPT。 我们使用 GPT-4 生成所有微调数据，并使用 GPT-3.5 和 GPT-4 进行提示。 截至 2023 年 11 月，两个模型均以 ChatCompletion 模式运行，温度设为 0.5。
Qwen-14B-Chat. An open-source conversational model from Alibaba Cloud, featuring 14 billion parameters, which has demonstrated exceptional performance in tool utilization. Qwen-14B-Chat was used for both fine-tuning and prompting. The parameter configuration for the Supervised Fine-Tuning (SFT) is detailed in Table 1. The hyperparameter settings for the prompting and fine-tuning methods during inference are identical, also shown in Table 1. In the inference phase, we utilized an NVIDIA A100 GPU equipped with 80GB of memory, offering robust computational power and substantial memory capacity for efficient processing.
Qwen-14B-Chat。 这是阿里云推出的开源对话模型，具有 140 亿参数，并在工具使用方面展现出卓越表现。 Qwen-14B-Chat 同时用于微调和提示。 监督微调（SFT）的参数配置详见表1。 提示方法与微调方法在推理期间使用相同的超参数设置，同样见表1。 在推理阶段，我们使用配备 80GB 显存的 NVIDIA A100 GPU，其强大计算能力和充足显存可支持高效处理。
Another distinction between prompting and fine-tuning methods is the use of one-shot guidance in prompting for structured output generation, whereas fine-tuning omits this step. Complete prompts for various architectures are detailed in the appendix.
提示方法与微调方法的另一个区别是：提示方法使用单样本引导来生成结构化输出，而微调省略了这一步。 不同架构的完整提示详见附录。
| SFT Hyper-parameters | |
|---|---|
| Hyper parameter | Value |
| precision | bfloat16 |
| model_max_length | 4096 |
| epochs | 3 |
| batch size | 64 |
| learning rate | 5e-6 |
| warmup ratio | 0.03 |
| LR scheduler type | cosine |
| Inference Hyper-parameters | |
|---|---|
| Hyper parameter | Value |
| max_new_tokens | 300 |
| top_p | 0.85 |
| temperature | 0.5 |
| repetition_penalty | 1.1 |
4.4 Evaluation
In the challenging landscape of human-computer dialogue systems, the evaluation of agent performance necessitates a nuanced approach. This is particularly pertinent when agents are tasked with engaging in direct conversations with human users, where the ultimate goal is to nurture a trust-based relationship. To achieve this, agents must exhibit a spectrum of qualities: they must be not only helpful and trustworthy but also responsive in a timely manner, and capable of understanding and articulating responses in a variety of contexts, akin to human interaction. This paper delineates seven sophisticated metrics designed to rigorously assess both the quality and efficiency of agent responses. The specific metrics and their corresponding scoring criteria are detailed in Table 2.
在人机对话系统这一复杂领域中，评估智能体性能需要采用细致的方法。 当智能体需要与人类用户直接对话，并以培养基于信任的关系为最终目标时，这一点尤其重要。 为实现这一目标，智能体必须展现一系列品质：不仅要乐于助人且值得信赖，还要及时响应，并能像人类交互一样在各种上下文中理解并清楚表达回答。 本文提出七项精细指标，用于严格评估智能体响应的质量和效率。 具体指标及其相应评分标准详见表2。
| Dimension | Metric | Score | Description |
|---|---|---|---|
| Quality | Specificity | 0 | Vague, general answer without specific information or details. |
| 1 | Provides some specifics, but lacks detail or full relevance to the question. | ||
| 2 | Directly addressing the user’s query with detailed and specific information. | ||
| Factuality | 0 | Contains false information, clearly contradicts facts. | |
| 1 | Mostly accurate, with minor inaccuracies or oversights. | ||
| 2 | Completely accurate, all information is fact-checked. | ||
| Coherence | 0 | Logically disorganized, unrelated to prior content or overall topic. | |
| 1 | Generally coherent, with some logical inconsistencies. | ||
| 2 | Very coherent, logically sound, closely aligned with the conversation topic. | ||
| Naturalness | 0 | Mechanical and unnatural, deviating from human conversational norms. | |
| 1 | Imitates natural dialogue to an extent, but still somewhat stiff or unnatural. | ||
| 2 | Smooth and natural, akin to human dialogue, easily understood and accepted. | ||
| Efficiency | Plan Steps | - | Number of planning steps. |
| Action Steps | - | Number of action steps. | |
| Inference Speed | - | The average time taken to process each user query, measured in seconds. |
For assessing quality, we leverage human-centric annotation methods that closely replicate human evaluative standards. Meanwhile, the efficiency metrics are derived through a systematic statistical analysis, providing concrete, quantifiable insights.
在质量评估方面，我们采用以人为中心的标注方法，紧密复现人类评估标准。 与此同时，效率指标通过系统化统计分析得到，提供具体且可量化的洞察。
4.5 Ablation Study
To demonstrate the effectiveness of the RAISE framework and fine-tuning method proposed in this paper, we conduct several ablation experiments in this section. The experiments are divided into two main aspects: (1) Comparative Analysis of Different Frameworks: We evaluate the performance of various frameworks under the same capability activation method, comparing their results using both the prompting and fine-tuning methods. The evaluation results are presented in Table 3. (2) Comparative Analysis of Different Capability Activation Methods: We compare the performance of the prompting method and fine-tuning method within the same framework, with the evaluation results presented in Table 4.
为了证明本文提出的 RAISE 框架和微调方法的有效性，本节开展了多项消融实验。 实验分为两个主要方面：(1) 不同框架的比较分析：我们在相同能力激活方法下评估不同框架的性能，并同时使用提示方法和微调方法比较其结果。 评估结果见表3。 (2) 不同能力激活方法的比较分析：我们比较同一框架内提示方法和微调方法的性能，评估结果见表4。
It's important to note that the inference speed of the OpenAI GPT API is subject to platform and network variations, so this metric was omitted from our analysis. The inference environment for Qwen-14B-Chat was kept consistent, utilizing an A100 GPU with 80GB memory.
需要注意的是，OpenAI GPT API 的推理速度会受到平台和网络变化的影响，因此我们的分析省略了该指标。 Qwen-14B-Chat 的推理环境保持一致，使用配备 80GB 显存的 A100 GPU。
| Framework | Spec. | Fact. | Coher. | Nat. | Ov. Qual. Score | Plan Steps | Act. Steps | Inf. Speed(s) |
|---|---|---|---|---|---|---|---|---|
| Prompting (GPT-4) | ||||||||
| Act-Only | 1.89 | 1.66 | 1.95 | 1.87 | 7.37 | - | 1.29 | - |
| ReAct | 1.98 | 1.87 | 1.93 | 1.79 | 7.57 | 2 | 1 | - |
| ReAct+Scratchpad | 1.98 | 1.88 | 1.99 | 1.65 | 7.5 | 1.97 | 0.96 | - |
| ReAct+Examples | 1.96 | 1.87 | 1.96 | 1.93 | 7.72 | 2.1 | 1.1 | - |
| RAISE | 1.95 | 1.92 | 1.97 | 1.85 | 7.69 | 1.79 | 0.8 | - |
| Fine-tuning (Qwen-14B-Chat) | ||||||||
| Act-Only | 1.66 | 1.71 | 1.82 | 1.92 | 7.11 | - | 0.66 | 1.935 |
| ReAct | 1.88 | 1.79 | 1.93 | 1.92 | 7.52 | 1.88 | 0.88 | 4.315 |
| ReAct+Scratchpad | 1.91 | 1.81 | 1.93 | 1.96 | 7.61 | 1.6 | 0.61 | 3.833 |
| ReAct+Examples | 1.93 | 1.82 | 1.96 | 1.95 | 7.66 | 1.33 | 0.33 | 3.327 |
| RAISE | 1.87 | 1.9 | 1.96 | 1.98 | 7.71 | 1.26 | 0.26 | 3.227 |
| Method | Spec. | Fact. | Coher. | Nat. | Ov. Qual. Score | Plan Steps | Act. Steps |
|---|---|---|---|---|---|---|---|
| RAISE | |||||||
| Prompting (GPT-3.5) | 1.65 | 1.72 | 1.66 | 1.67 | 6.7 | 2.13 | 5 |
| Prompting (Qwen-14B-Chat) | 1.69 | 1.66 | 1.68 | 1.65 | 6.68 | 2.06 | 1.2 |
| Prompting (GPT-4) | 1.95 | 1.92 | 1.97 | 1.85 | 7.69 | 1.79 | 0.8 |
| Fine-tuning (Qwen-14B-Chat) | 1.87 | 1.9 | 1.96 | 1.98 | 7.71 | 1.26 | 0.26 |
| ReAct+Scratchpad | |||||||
| Prompting (GPT-3.5) | 1.62 | 1.57 | 1.74 | 1.55 | 6.48 | 2.19 | 1.18 |
| Prompting (Qwen-14B-Chat) | 1.68 | 1.56 | 1.71 | 1.7 | 6.65 | 2.07 | 1.09 |
| Prompting (GPT-4) | 1.98 | 1.88 | 1.99 | 1.65 | 7.5 | 1.97 | 0.96 |
| Fine-tuning (Qwen-14B-Chat) | 1.91 | 1.81 | 1.93 | 1.96 | 7.61 | 1.6 | 0.61 |
Upon analyzing the experimental outcomes, the following key conclusions emerge with respect to the efficacy and efficiency of different agent frameworks and methods:
通过分析实验结果，我们就不同智能体框架和方法的有效性与效率得出以下关键结论：
Framework Performance Ranking within the Same LLM. The RAISE framework demonstrates superior performance, followed by ReAct+Examples, ReAct+Scratchpad, ReAct, and lastly, the Act-Only approach. This ranking indicates a clear gradient in effectiveness and efficiency, highlighting the incremental benefits of integrating additional elements like examples and scratchpads into the base ReAct model.
同一 LLM 内的框架性能排名。 RAISE 框架表现最佳，其后依次是 ReAct+Examples、ReAct+Scratchpad、ReAct，最后是 Act-Only 方法。 这一排名显示出有效性和效率上的明显梯度，突出了在基础 ReAct 模型中加入示例和草稿区等额外元素所带来的递增收益。
Comparative Analysis of Capability Activation Methods within Identical Frameworks. The fine-tuning approach outperforms the prompting method. This suggests that tailored training and customization of models to specific tasks or datasets result in more efficient and effective performance compared to using generalized prompt-based interactions.
相同框架内能力激活方法的比较分析。 微调方法优于提示方法。 这表明，与使用通用的基于提示的交互相比，针对特定任务或数据集对模型进行定制训练和调整能够带来更高效、更有效的性能。
In the following parts, we delve into detailed analyses of these findings, examining the implications and potential applications of each framework and methodology.
下面我们将深入分析这些发现，考察各框架和方法的影响及潜在应用。
Chain of Thought (CoT): A Catalyst for Enhanced Comprehension and Response Accurac. CoT significantly boosts AI's ability to deeply comprehend and precisely respond to complex queries. Our experimental findings reaffirm the importance of CoT in complex tasks. For instance, in comparative experiments across different frameworks, agents employing the Act-Only method showed substantially lower performance compared to those incorporating CoT. These findings underscore the critical role of CoT in promoting AI models to deliver depth-oriented and logically coherent responses, particularly in scenarios requiring complex reasoning.
思维链（CoT）：增强理解能力和回答准确性的催化剂。 CoT 显著提升了 AI 深入理解复杂问题并作出精确回答的能力。 我们的实验结果再次证实了 CoT 在复杂任务中的重要性。 例如，在不同框架的对比实验中，采用 Act-Only 方法的智能体性能明显低于引入 CoT 的智能体。 这些发现凸显了 CoT 在促进 AI 模型给出深入且逻辑连贯回答方面的关键作用，尤其是在需要复杂推理的场景中。
RAISE Architecture: Dual Benefits of Efficiency from Scratchpad and Examples. The RAISE architecture, by harmonizing Scratchpad and Example mechanisms, attains a dual advantage in processing efficiency and output quality. The application of Scratchpad significantly enhances the efficiency in handling complex tasks, while the utilization of Examples simultaneously bolsters the response's naturalness, specificity, and efficiency. This dual advantage positions the RAISE architecture as a suitable choice for scenarios demanding rapid, accurate, and naturally interactive responses.
RAISE 架构：草稿区与示例带来的双重效率收益。 RAISE 架构通过协调草稿区和示例机制，在处理效率与输出质量方面取得双重优势。 草稿区的应用显著提升了处理复杂任务的效率，而示例的使用同时增强了回答的自然度、具体性和效率。 这种双重优势使 RAISE 架构适用于要求快速、准确且自然交互式回答的场景。
Fine-tuning: A Lever for Enhancing Agent Performance. Utilizing diverse, high-quality datasets for fine-tuning helps to better align AI models with human behavioral logic, potentially leading to notable improvements in specific application areas. This approach excels in specialized tasks, offering a high degree of professionalism and customization. For instance, in the RAISE framework under fine-tuning, the overall quality score reached 7.71, and inference efficiency was optimized. These results validate the effectiveness of fine-tuning in delivering precise, human-like and efficient outcomes, particularly suitable for scenarios requiring customized solutions, such as online real estate services.
微调：提升智能体性能的杠杆。 使用多样化的高质量数据集进行微调，有助于让 AI 模型更好地与人类行为逻辑对齐，并可能在特定应用领域带来显著提升。 这种方法在专业任务中表现出色，能够提供高度的专业性和定制能力。 例如，在采用微调的 RAISE 框架中，总体质量得分达到 7.71，同时推理效率也得到优化。 这些结果验证了微调在提供精确、类人且高效结果方面的有效性，尤其适用于在线房地产服务等需要定制解决方案的场景。
Fine-tuning: Enhancing Cost-Efficiency and Speed during Agent Inference. Although fine-tuning may require higher initial investments, its long-term benefits in operational efficiency and precision can offset these costs. In application, fine-tuned models often necessitate fewer computational resources, thereby reducing operational costs and accelerating response times. This cost-effectiveness, coupled with improved performance, makes fine-tuning a prudent choice for specific, resource-intensive tasks.
微调：提升智能体推理期间的成本效益和速度。 尽管微调可能需要较高的初始投入，但其在运行效率和精度方面的长期收益能够抵消这些成本。 在实际应用中，微调模型通常需要更少的计算资源，从而降低运行成本并缩短响应时间。 这种成本效益与性能提升相结合，使微调成为特定资源密集型任务中的审慎选择。
Strategic Deployment of Language Agents: When to Choose Fine-tuning Over Prompting. The decision to opt for fine-tuning or prompting hinges on the specific requirements of the application. Fine-tuning offers superior performance and efficiency in specialized domains but may involve higher initial costs and training needs. In contrast, prompting is more flexible in handling a wide range of queries but may slightly lag behind fine-tuned models in specificity and stability. Strategic decision-making in this context involves balancing these factors against the specific needs of the application, budget constraints, and performance expectations.
语言智能体的战略部署：何时选择微调而非提示。 选择微调还是提示取决于应用的具体需求。 微调在专业领域中提供更优的性能和效率，但可能涉及更高的初始成本与训练需求。 相比之下，提示在处理广泛问题时更加灵活，但在具体性和稳定性方面可能略逊于微调模型。 在这一情境下作出战略决策，需要根据应用的具体需求、预算约束和性能预期来权衡这些因素。
5. Related Work
The exploration and advancements in AI agents have captivated the AI research community for some time. Defined as artificial entities capable of perceiving their surroundings, making decisions, and executing actions, AI agents represent a significant stride in artificial intelligence.
长期以来，AI 智能体的探索与进展一直吸引着人工智能研究界的关注。 AI 智能体被定义为能够感知周围环境、作出决策并执行行动的人工实体，代表着人工智能领域的一项重大进步。
The advent of Large Language Models (LLMs) has been a pivotal development, often regarded as a step towards the realization of Artificial General Intelligence (AGI). In recent years, there has been an influx of studies proposing intricate LLM-based architectures for AI agents. These architectures are crucial in enabling agents to navigate complex dialogue scenarios and effectively apply their acquired knowledge.
大语言模型（LLM）的出现是一项关键进展，通常被视为迈向通用人工智能（AGI）的一步。 近年来，大量研究为 AI 智能体提出了复杂的 LLM 架构。 这些架构对于智能体应对复杂对话场景并有效运用所获取的知识至关重要。
This body of work primarily revolves around two core aspects:
这类工作主要围绕两个核心方面展开：
- Planning: Central to the functionality of dialogue agents is the concept of Chain-of-Thought (CoT) reasoning. This involves eliciting logical rationales via CoT prompts. However, integrating this reasoning effectively into dialogues remains challenging. The ReAct framework presents an approach that guides LLMs in reasoning before planning actions, addressing this issue.
- Tool Use: Another critical facet is the ability of LLMs to utilize external tools and resources. Studies have demonstrated the proficiency of LLMs in leveraging external tools and APIs. Moreover, the capacity to extract and integrate knowledge from external sources has been further exemplified by projects like WebGPT and ExpeL.
- 规划： 对话智能体功能的核心是思维链（CoT）推理。这涉及通过 CoT 提示引出逻辑推理过程。然而，如何将这种推理有效融入对话仍具有挑战。ReAct 框架提出了一种方法，引导 LLM 在规划行动之前进行推理，从而解决这一问题。
- 工具使用： 另一个关键方面是 LLM 使用外部工具和资源的能力。已有研究证明了 LLM 利用外部工具和 API 的熟练程度。此外，WebGPT 和 ExpeL 等项目进一步展示了从外部来源提取并整合知识的能力。
In addition to these areas, several works have focused on broader algorithmic frameworks for LLM-based agents. On the other hand, specific dialogue agents have also been a focal point. The fine-tuning of LLMs within agents is another critical area, with works exploring this aspect.
除这些领域外，一些工作还关注面向 LLM 智能体的更广泛算法框架。 另一方面，特定的对话智能体也一直是研究重点。 智能体内部 LLM 的微调是另一个关键领域，已有工作对此进行了探索。
These developments underscore the growing complexity and capabilities of LLM-based AI agents, highlighting both the challenges and the innovations shaping the field.
这些进展凸显了基于 LLM 的 AI 智能体日益增长的复杂性和能力，同时也揭示了塑造这一领域的挑战与创新。
6. Conclusions and Future work
This study introduces RAISE, an advanced architecture enhancing Long Language Models (LLMs) like GPT-4 for conversational agents. Building on the ReAct framework, RAISE integrates a dual-component memory system, improving dialogue context retention and continuity. We also propose a fine-tuning method within RAISE, which enhances agent controllability and efficiency, particularly in real estate sales, though applicable in various domains.
本研究提出 RAISE，这是一种用于增强 GPT-4 等大型语言模型（LLM）在对话智能体中应用的先进架构。 RAISE 以 ReAct 框架为基础，集成了双组件记忆系统，从而改善对话上下文的保留与连续性。 我们还在 RAISE 中提出一种微调方法，用于增强智能体的可控性和效率；该方法在房地产销售中尤其有效，同时也可应用于多个领域。
However, the study has limitations, including potential hallucination issues and challenges in handling complex logic problems, necessitating further research. Despite these limitations, RAISE presents a promising advancement in adaptable, context-aware conversational agents, offering a foundation for future developments in artificial intelligence.
然而，本研究仍存在局限，包括潜在的幻觉问题以及处理复杂逻辑问题时面临的挑战，因此还需要进一步研究。 尽管存在这些局限，RAISE 仍代表了可适应、具备上下文感知能力的对话智能体方面的一项可喜进展，并为人工智能的未来发展奠定基础。