---

ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory

MemoryAgentDatabaseSQL600+160+清华大学北京智源人工智能研究院浙江大学

Hu C, Fu J, Du C, et al. ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory[J]. arXiv preprint arXiv:2306.03901v2, 2023.

https://chatdatabase.github.io/

https://github.com/huchenxucs/ChatDB

---

ChatDB：用数据库作为符号记忆增强 LLM

Abstract

Large language models (LLMs) with memory are computationally universal. However, mainstream LLMs are not taking full advantage of memory, and the designs are heavily influenced by biological brains. Due to their approximate nature and proneness to the accumulation of errors, conventional neural memory mechanisms cannot support LLMs to simulate complex reasoning. In this paper, we seek inspiration from modern computer architectures to augment LLMs with symbolic memory for complex multi-hop reasoning. Such a symbolic memory framework is instantiated as an LLM and a set of SQL databases, where the LLM generates SQL instructions to manipulate the SQL databases. We validate the effectiveness of the proposed memory framework on a synthetic dataset requiring complex reasoning.

带有记忆的大语言模型（LLM）在计算上是通用的。 然而，主流 LLM 并没有充分利用记忆，而且其设计深受生物大脑的影响。 由于传统神经记忆机制具有近似性，并且容易累积错误，它们无法支持 LLM 模拟复杂推理。 在本文中，我们从现代计算机架构中获得启发，用符号记忆增强 LLM，以支持复杂的多跳推理。 这样一个符号记忆框架被实例化为一个 LLM 和一组 SQL 数据库，其中 LLM 生成 SQL 指令来操作 SQL 数据库。 我们在一个需要复杂推理的合成数据集上验证了所提出记忆框架的有效性。

图1：ChatDB 的整体工作流程。LLM 控制器控制对记忆的读写操作。记忆存储历史信息，并提供相关历史信息来辅助响应用户输入。在 ChatDB 中，我们关注用数据库作为 LLM 的符号记忆来增强它们。

1. Introduction

Large language models (LLMs), such as GPT-4 and PaLM 2, have increasingly become an essential component of modern artificial intelligence (AI) systems, revolutionizing our understanding of natural language processing (NLP) and transforming various industries. While LLMs have made significant strides in understanding and generating contextually relevant responses, they have limitations. One of the main challenges is that multi-turn interactions with language models generate a large number of tokens, which can easily exceed the input token limit of LLMs. For example, GPT-4 (32K) can only handle 32,000 tokens. As the interaction progresses, the LLMs must maintain contextual information (e.g., user inputs, and previous responses) and generate responses based on the accumulated data. Simply concatenating all contextual information and cramming it into LLMs, however, can easily exceed the processing capabilities of LLMs and accumulate errors, causing the model to lose track of the conversation and produce less accurate responses.

大型语言模型（LLM），例如 GPT-4 和 PaLM 2，已经日益成为现代人工智能（AI）系统的重要组成部分，革新了我们对自然语言处理（NLP）的理解，并改变了多个行业。 尽管 LLM 在理解并生成与上下文相关的回复方面取得了显著进展，它们仍然存在局限。 一个主要挑战是，与语言模型进行多轮交互会产生大量 token，很容易超过 LLM 的输入 token 限制。 例如，GPT-4（32K）只能处理 32,000 个 token。 随着交互推进，LLM 必须维护上下文信息（例如用户输入和先前回复），并基于累积数据生成回答。 然而，简单地拼接所有上下文信息并塞入 LLM，很容易超出 LLM 的处理能力并累积错误，导致模型失去对话线索并生成较不准确的回复。

Some neural memory mechanisms have been explored to overcome the limited token input issue of LLMs. The memory components serve as a storage and retrieval system for relevant information from previous interactions. However, augmenting LLMs with conventional neural memory usually leads to difficulties in storing, retrieving, and manipulating historical information in memory, especially for tasks requiring complex multi-hop reasoning. Two main causes are (a) They do not store historical information in a structured form; (b) Their manipulation of the information stored in memory is not symbolic, as they all rely on some vector similarity calculations, which can be inaccurate, thus leading to the accumulation of errors.

一些神经记忆机制已经被探索，用于克服 LLM 输入 token 受限的问题。 记忆组件作为存储和检索系统，用于保存并取回先前交互中的相关信息。 然而，用传统神经记忆增强 LLM 通常会在存储、检索和操作记忆中的历史信息时遇到困难，尤其是在需要复杂多跳推理的任务中。 主要原因有两点：（a）它们没有以结构化形式存储历史信息；（b）它们对记忆中信息的操作不是符号化的，因为它们都依赖某种向量相似度计算，而这可能不准确，从而导致错误累积。

To address the aforementioned issues, we propose using databases as novel symbolic memory for LLMs. The whole framework is named ChatDB. As shown in Figure 1, ChatDB consists of two components: an LLM controller and its memory. The LLM controller can be any commonly used LLM and is responsible for controlling the read and write operations to the memory. The memory of LLMs, which can be symbolic or non-symbolic, or a combination of both, is responsible for storing historical information and providing information when needed to assist the LLM in responding to user input. In ChatDB, we focus on using databases as symbolic memory, which allows for the structured storage of historical information through the execution of a symbolic language, namely SQL statements. These SQL statements are generated by an LLM. Incorporating a database as symbolic memory is particularly useful in scenarios requiring precise recording, modification, querying, deletion, and analysis of historical data. For example, a store manager needs to maintain daily sales records, where using plain text or matrices as memory is unsuitable. However, using a database as external symbolic memory is highly suitable. The database enables accurate operations, including data insertion, deletion, update, and selection, using SQL statements. Thus, employing databases as external symbolic memory ensures precision and efficiency in managing and manipulating historical data, significantly enhancing the performance of LLMs in scenarios that require high accuracy and long-term data recording and processing.

为了解决上述问题，我们提出使用数据库作为 LLM 的新型符号记忆。 整个框架命名为 ChatDB。 如图1所示，ChatDB 由两个组件组成：一个 LLM 控制器及其记忆。 LLM 控制器可以是任何常用 LLM，并负责控制对记忆的读写操作。 LLM 的记忆可以是符号化的、非符号化的，或二者的组合；它负责存储历史信息，并在需要时提供信息，以辅助 LLM 响应用户输入。 在 ChatDB 中，我们关注将数据库作为符号记忆，这允许通过执行符号语言，即 SQL 语句，对历史信息进行结构化存储。 这些 SQL 语句由 LLM 生成。 将数据库作为符号记忆加入系统，在需要精确记录、修改、查询、删除和分析历史数据的场景中特别有用。 例如，商店经理需要维护每日销售记录，此时使用纯文本或矩阵作为记忆并不合适。 然而，使用数据库作为外部符号记忆非常合适。 数据库能够通过 SQL 语句准确执行数据插入、删除、更新和选择等操作。 因此，采用数据库作为外部符号记忆，能够确保管理和操作历史数据时的精确性和效率，并在需要高准确率以及长期数据记录和处理的场景中显著提升 LLM 的性能。

In the ChatDB framework, we propose the chain-of-memory (CoM) approach to manipulate the external symbolic memory more effectively, thereby further enhancing the reasoning capabilities of LLMs. The chain-of-memory approach transforms user input into a series of intermediate memory operation steps that lead to final results. Through the chain-of-memory approach, a complex problem is decomposed into multiple steps of memory operations, significantly reducing the complexity of problem-solving. In ChatDB, each intermediate step involves one or more SQL statements.

在 ChatDB 框架中，我们提出 chain-of-memory（CoM）方法，以更有效地操作外部符号记忆，从而进一步增强 LLM 的推理能力。 chain-of-memory 方法把用户输入转换为一系列通向最终结果的中间记忆操作步骤。 通过 chain-of-memory 方法，复杂问题被分解为多个记忆操作步骤，显著降低了解决问题的复杂度。 在 ChatDB 中，每个中间步骤包含一个或多个 SQL 语句。

Our ChatDB makes several contributions to the field of LLMs. Firstly, we propose augmenting LLMs with databases as their external symbolic memory, allowing for structured storage of historical data and enabling symbolic and complex data operations using SQL statements. Secondly, our chain-of-memory approach enables effective memory manipulation by converting user input into multi-step intermediate memory operations, which enhance the performance of ChatDB, enabling it to handle complex, multi-table database interactions with improved accuracy and stability. Finally, our experiments demonstrate that augmenting LLMs with symbolic memory improves multi-hop reasoning capabilities and prevents error accumulation, thereby enabling ChatDB to significantly outperform ChatGPT on a synthetic dataset.

我们的 ChatDB 对 LLM 领域作出了几项贡献。 首先，我们提出用数据库作为外部符号记忆来增强 LLM，使历史数据能够结构化存储，并支持使用 SQL 语句进行符号化和复杂的数据操作。 其次，我们的 chain-of-memory 方法通过把用户输入转换为多步中间记忆操作来实现有效记忆操作，这提升了 ChatDB 的性能，使其能够以更高准确性和稳定性处理复杂的多表数据库交互。 最后，我们的实验表明，用符号记忆增强 LLM 能够提升多跳推理能力并防止错误累积，从而使 ChatDB 在一个合成数据集上显著优于 ChatGPT。

2. Related Work

Memory-Augmented Large Language Models (LLMs). LLMs, such as GPT-4 and PaLM 2, have demonstrated powerful reasoning and decision-making abilities. However, LLMs are often hindered by their limited context window sizes (e.g., GPT-4 can only handle 32K tokens). Memory-augmented LLMs incorporate a memory module that prevents the model from forgetting crucial information and allows it to handle long text inputs that exceed the context window size. Retrieval-augmented in-context learning uses retrieval models (RM) to retrieve relevant information that can be inserted into the LLM as a prompt. For example, Auto-GPT and Generative Agents utilize a memory module to store the text prompt directly, allowing the agent to keep track of its history. The past and current prompts are then input into the LLM for processing. Neural Turing Machines (NMT), which incorporate the recurrent neural network (RNN) with external trainable memory resources and learn to interact with the memory module with gradient descent. Gated Graph Sequence Neural Network (GGS-NN) constructs and modifies graphs and utilizes the graphs to produce reasonable outputs. Recurrent Memory Transformer (RMT) introduces additional memory tokens to the input and output sequences to store, process and exchange local and global information between segments of long sequences, and then train the model to control both memory operation and sequence representations processing.

记忆增强大型语言模型（LLM）。 LLM，例如 GPT-4 和 PaLM 2，已经展示出强大的推理和决策能力。 然而，LLM 往往受限于其有限的上下文窗口大小（例如 GPT-4 只能处理 32K token）。 记忆增强 LLM 引入记忆模块，防止模型忘记关键信息，并使其能够处理超过上下文窗口大小的长文本输入。 检索增强的上下文学习使用检索模型（RM）检索相关信息，并将其作为提示插入 LLM。 例如，Auto-GPT 和 Generative Agents 使用记忆模块直接存储文本提示，使智能体能够跟踪自己的历史。 随后，过去和当前的提示会被输入 LLM 进行处理。 Neural Turing Machines（NMT）将循环神经网络（RNN）与外部可训练记忆资源结合，并通过梯度下降学习与记忆模块交互。 Gated Graph Sequence Neural Network（GGS-NN）构建并修改图，并利用这些图产生合理输出。 Recurrent Memory Transformer（RMT）向输入和输出序列引入额外记忆 token，用于在长序列片段之间存储、处理和交换局部与全局信息，并训练模型同时控制记忆操作和序列表示处理。

Reasoning with LLMs. LLMs are known to struggle in complex reasoning tasks. Previous methods focus on incorporating specially designed supervisory signals or fine-tuning to enhance the reasoning ability of language models. Recent methods mainly improve the reasoning ability of language models through In-Context Learning. The most representative of these is Chain-of-Thought (CoT), which presents the intermediate reasoning process of solving sample problems to the language model, greatly enhancing its reasoning capabilities.

用 LLM 进行推理。 众所周知，LLM 在复杂推理任务中会遇到困难。 先前方法主要通过引入专门设计的监督信号或微调来增强语言模型的推理能力。 近期方法主要通过上下文学习提升语言模型的推理能力。 其中最具代表性的是 Chain-of-Thought（CoT），它向语言模型展示解决样例问题的中间推理过程，从而大幅增强其推理能力。

LLMs with DBs. LLMs have demonstrated an impressive capability in generating code, including Python code, execution commands for Excel, and Structured Query Language (SQL) for databases. ChatExcel uses LLMs to generate the Excel execution command, simplifying the user interaction process. BINDER proposes a framework that maps task inputs to executable programs in a programming language (e.g., Python code) bound with an API to call LLMs to perform a wide range of functionalities. SQL-PALM proposes an LLM-based Text-to-SQL model, using the execution-based self-consistent prompting approach, and outperforms previous Text-2-SQL methods by a large margin. While previous works involve databases to some extent, our proposed ChatDB system differs significantly from these methods. In specific, ChatDB views the databases as the external symbolic memory module for the LLM, and then leverages the database for reading and writing essential data information to enhance the reasoning process via chain-of-memory, leading to more accurate reasoning results.

结合数据库的 LLM。 LLM 已经展示出令人印象深刻的代码生成能力，包括 Python 代码、Excel 执行命令，以及面向数据库的结构化查询语言（SQL）。 ChatExcel 使用 LLM 生成 Excel 执行命令，从而简化用户交互过程。 BINDER 提出一个框架，将任务输入映射为编程语言（例如 Python 代码）中的可执行程序，并绑定一个 API 来调用 LLM 执行广泛功能。 SQL-PALM 提出一种基于 LLM 的 Text-to-SQL 模型，使用基于执行的自一致提示方法，并大幅优于先前的 Text-2-SQL 方法。 虽然先前工作在某种程度上涉及数据库，但我们提出的 ChatDB 系统与这些方法有显著差异。 具体而言，ChatDB 将数据库视为 LLM 的外部符号记忆模块，然后利用数据库读取和写入必要数据信息，通过 chain-of-memory 增强推理过程，从而得到更准确的推理结果。

Tool-using LLMs. From the tool-using perspective, ChatDB can also be seen as an LLM utilizing DBs as a tool. Toolformer, through a series of demonstrations, instructs the language model that it can invoke some APIs to utilize external tools to solve the current problem. Another representative work is Auto-GPT, which enables the language models to complete a series of impressive tasks using a search engine. The advantage of ChatDB, which uses a database as an external tool, is that it allows the language model to maintain more accurate records and use historical data, thereby addressing more complex problems, especially those that require accurate historical data for reasoning.

使用工具的 LLM。 从工具使用的角度看，ChatDB 也可以被视为一个把数据库作为工具来使用的 LLM。 Toolformer 通过一系列示例告诉语言模型，它可以调用一些 API 来利用外部工具解决当前问题。 另一个代表性工作是 Auto-GPT，它使语言模型能够使用搜索引擎完成一系列令人印象深刻的任务。 ChatDB 使用数据库作为外部工具，其优势在于它允许语言模型维护更准确的记录并使用历史数据，从而处理更复杂的问题，尤其是那些需要准确历史数据进行推理的问题。

3. ChatDB

In this section, we first briefly introduce the task definition and setting. We then describe the overall framework of our proposed ChatDB. Finally, we delve into the details of the chain-of-memory approach, which is the main component of ChatDB.

在本节中，我们首先简要介绍任务定义和设置。 随后，我们描述所提出 ChatDB 的整体框架。 最后，我们深入介绍 chain-of-memory 方法的细节，这是 ChatDB 的主要组成部分。

3.1. Task Definition

Given a user input in natural language and details of existing tables in the database (not required if there are no existing tables), the goal is to manipulate the symbolic memory, i.e., the external database, to fulfill the user's request. For instance, if the user (e.g., a store manager) command is to record, modify, query, and delete specific data, the corresponding SQL operation should be to insert, update, select, and delete the relevant data in the appropriate tables, respectively. These operations often involve multiple tables in the database.

给定自然语言形式的用户输入，以及数据库中已有表的详细信息（如果没有已有表则不需要），目标是操作符号记忆，即外部数据库，以满足用户请求。 例如，如果用户（例如商店经理）的命令是记录、修改、查询和删除特定数据，那么相应 SQL 操作应分别是在适当表中插入、更新、选择和删除相关数据。 这些操作通常涉及数据库中的多个表。

图2：ChatDB 框架。红色箭头线表示 chain-of-memory 的流程，表明多个记忆操作之间的连接。数据库表之间的红色箭头线表示主键和外键之间的引用关系，从主键指向外键。为简洁起见，每个表仅显示前四列。该示例展示了手机号为 823451 的客户退回 2023-01-02 购买商品的过程。

3.2. Framework Overview

ChatDB framework consists of three main stages: input processing, chain-of-memory, and response summary, as illustrated in Figure 2. Algorithm 1 provides a detailed illustration of the entire algorithmic procedure of ChatDB to respond to the user input.

ChatDB 框架由三个主要阶段组成：输入处理、chain-of-memory 和响应总结，如图2所示。 算法1 详细展示了 ChatDB 响应用户输入的完整算法流程。

Input Processing. ChatDB generates a series of intermediate steps to manipulate the symbolic memory by utilizing LLMs if responding to the user input requires the use of memory. Otherwise, we use LLMs directly to generate a reply. Chain-of-Memory. ChatDB executes a series of intermediate memory operation steps to interact with symbolic memory. ChatDB manipulates the symbolic memory in sequence according to a series of previously generated SQL statements, including operations such as insert, update, select, delete, etc. The external database executes the corresponding SQL statements, updates the database, and returns the results. It is worth noting that ChatDB decides whether to update the memory operation step based on the results of previous SQL statements before executing this operation. ChatDB executes the next step following the same procedure until all operations on the memory are completed. Response Summary. ChatDB summarizes the final response to the user based on the results of a series of chain-of-memory steps.

输入处理。 如果响应用户输入需要使用记忆，ChatDB 会利用 LLM 生成一系列中间步骤来操作符号记忆。 否则，我们直接使用 LLM 生成回复。 Chain-of-Memory。 ChatDB 执行一系列中间记忆操作步骤，以与符号记忆交互。 ChatDB 根据先前生成的一系列 SQL 语句，按顺序操作符号记忆，包括插入、更新、选择、删除等操作。 外部数据库执行对应 SQL 语句、更新数据库并返回结果。 值得注意的是，ChatDB 在执行该操作之前，会基于先前 SQL 语句的结果决定是否更新记忆操作步骤。 ChatDB 按照同样流程执行下一步，直到完成对记忆的所有操作。 响应总结。 ChatDB 基于一系列 chain-of-memory 步骤的结果，向用户总结最终响应。

Algorithm 1: The algorithm of ChatDB

Input: userInput, dataBase
Output: reply

1. // Begin of Input Processing
2. if need manipulate memory to respond to userInput then
3. $$ memOps $\leftarrow {\mathrm{LLM}}_{\mathrm{getSteps}}(\text{userInput})$ // Generate intermediate steps using LLMs
4. else
5. $$ reply $\leftarrow \mathrm{LLM}(\text{userInput})$ // Generate reply using LLMs directly
6. $$ return reply
7. end if
8. // Begin of Chain-of-Memory
9. sqlResults $\leftarrow$ []
10. newMemOps $\leftarrow$ []
11. for each memOp in memOps do
12. $$ if need update memOp based on sqlResults then
13. $$ newMemOp $\leftarrow {\mathrm{LLM}}_{\mathrm{updateOperation}}(\text{memOp},\text{sqlResults})$
14. $$ else
15. $$ newMemOp $\leftarrow$ memOp
16. $$ end if
17. $$ sqlResult $\leftarrow \mathrm{executeOperation}(\text{newMemOp},\text{dataBase})$ // Execute operation on database
18. $$ sqlResults.append(sqlResult)
19. $$ newMemOps.append(newMemOp)
20. end for
21. // Begin of Response Summary
22. reply $\leftarrow {\mathrm{LLM}}_{\mathrm{summary}}(\text{userInput},\text{newMemOps},\text{sqlResults})$ // Summarize the final reply
23. return reply

3.3. Chain-of-Memory

Chain-of-thought emphasizes decomposing complex reasoning into a series of intermediate steps. Chain-of-memory (CoM) can be seen as a way to augment chain-of-thought by providing a symbolic memory mechanism to support the storage associated with those intermediate steps.

Chain-of-thought 强调将复杂推理分解为一系列中间步骤。 Chain-of-memory（CoM）可以被看作一种增强 chain-of-thought 的方式：它提供符号记忆机制，支持与这些中间步骤相关的存储。

The purpose of chain-of-memory is to enhance the reasoning capabilities and robustness of LLMs when manipulating symbolic memory. The approach involves converting the user input into a sequence of intermediate memory operations, enabling LLMs to more accurately and effectively manipulate the memory in a symbolic way. The ability to manipulate symbolic memory is particularly valuable for real-world applications that involve complex and accurate interactions with historical data, such as record-keeping and data analysis in a management setting.

chain-of-memory 的目标是在操作符号记忆时增强 LLM 的推理能力和鲁棒性。 该方法把用户输入转换为一系列中间记忆操作，使 LLM 能够以符号化方式更准确、更有效地操作记忆。 操作符号记忆的能力对于需要与历史数据进行复杂且精确交互的真实应用尤其有价值，例如管理场景中的记录保存和数据分析。

To enhance the performance and stability of our approach, we employ in-context learning, providing prompt exemplars of several sequences of chain-of-memory steps, and chain-of-thought prompting. A robust and accurate chain-of-memory process enables LLMs to better reason over symbolic memory and handle more complex scenarios.

为了增强我们方法的性能和稳定性，我们采用上下文学习，提供若干 chain-of-memory 步骤序列的提示示例，并使用 chain-of-thought 提示。 鲁棒且准确的 chain-of-memory 过程使 LLM 能够更好地基于符号记忆进行推理，并处理更复杂的场景。

The advantages of chain-of-memory are twofold. Firstly, it enables LLMs to perform complex database manipulations with greater accuracy, enhancing their multi-hop reasoning ability over symbolic memory. Secondly, by decomposing complex operations into a sequence of intermediate memory operations, the chain-of-memory approach enhances the capability of LLMs when handling complex, multi-table interactions. This approach enables LLMs to handle edge cases and unexpected scenarios better, making it a promising method for real-world applications.

chain-of-memory 的优势有两个方面。 首先，它使 LLM 能够更准确地执行复杂数据库操作，从而增强它们在符号记忆上的多跳推理能力。 其次，通过将复杂操作分解为一系列中间记忆操作，chain-of-memory 方法增强了 LLM 处理复杂多表交互时的能力。 该方法使 LLM 能够更好地处理边界情况和意外场景，因此是一种有前景的真实应用方法。

3.4. Comparison with previous memory-augmented LLMs

表1：与 prompt-based memory 和 matrix-based memory 的比较。

Type	Model	Memory format	Supported operations	Memory storage	Memory execution	Interpretability	State tracking
Symbolic	ChatDB	Symbolic storage (e.g., database)	Insert, Delete Update, Select	Structured	Symbolic	High	Yes
Prompt-based	Auto-GPT	Content and its vector embeddings	Insert, Select	Semi-Structured	Non-Symbolic	Normal	No
Matrix-based	RMT	Memory tokens/ memory matrices	Read, Write	Semi-Structured	Non-Symbolic	Low	Yes

In this subsection, we provide a comprehensive comparison between ChatDB and recent approaches that augment Transformer-based language models with memory module. The memory modules of language models proposed in previous work can be broadly categorized into two types. The first type of memory stores context and uses a retrieval model to find content from past interactions that's most relevant to the current conversation, and then uses it as a prompt for the language model. We refer to this type of memory as prompt-based memory. The second type of approach utilizes additional memory tokens or memory matrices as memory, which we refer to as matrix-based memory. We compare ChatDB with these approaches based on the following aspects:

在本小节中，我们对 ChatDB 与近期用记忆模块增强 Transformer 语言模型的方法进行全面比较。 先前工作提出的语言模型记忆模块大体可以分为两类。 第一类记忆存储上下文，并使用检索模型从过去交互中找到与当前对话最相关的内容，然后将其作为语言模型的提示。 我们将这类记忆称为 prompt-based memory。 第二类方法使用额外记忆 token 或记忆矩阵作为记忆，我们将其称为 matrix-based memory。 我们基于以下几个方面比较 ChatDB 与这些方法：

1. Memory Format. This aspect relates to the format used for storing memory. ChatDB utilizes databases as its memory. Prompt-based memory stores relevant interaction content and/or their corresponding vector embeddings. Matrix-based memory employs additional trainable memory tokens or trainable memory matrices.

1. 记忆格式。 这一方面涉及用于存储记忆的格式。 ChatDB 使用数据库作为其记忆。 Prompt-based memory 存储相关交互内容和/或其对应向量嵌入。 Matrix-based memory 使用额外可训练记忆 token 或可训练记忆矩阵。

2. Supported Operations. This aspect refers to the operations supported for manipulating memory. ChatDB supports operations such as insertion, deletion, update, and selection of data in the database memory. The prompt-based memory primarily supports insertion and selection operations but lacks complete support for updates and deletions. Matrix-based memory supports reading (selection) and writing (insertion, updating, deletion) operations. However, the exact operations performed by the neural network are not explicitly known.

2. 支持的操作。 这一方面指操作记忆时支持哪些操作。 ChatDB 支持在数据库记忆中对数据进行插入、删除、更新和选择等操作。 Prompt-based memory 主要支持插入和选择操作，但缺乏对更新和删除的完整支持。 Matrix-based memory 支持读取（选择）和写入（插入、更新、删除）操作。 然而，神经网络实际执行的确切操作并不是显式可知的。

3. Memory Storage. This aspect refers to the format in which data is stored in memory, specifically whether it is structured or not. ChatDB stores memory in a structured format using databases, while both the prompt-based memory and matrix-based memory are viewed as semi-structured. The reason is that vector embeddings and memory matrices have specific dimensions and sizes, but each individual dimension does not carry a specific and explicit meaning.

3. 记忆存储。 这一方面指数据在记忆中的存储格式，尤其是它是否结构化。 ChatDB 使用数据库以结构化格式存储记忆，而 prompt-based memory 和 matrix-based memory 都被视为半结构化。 原因是向量嵌入和记忆矩阵具有特定维度和大小，但每个单独维度并不承载特定且显式的含义。

4. Memory Execution. This aspect focuses on how memory operations are executed, specifically whether they are symbolic or not. ChatDB executes operations on its database memory using SQL, which is a symbolic language, thus making it inherently symbolic. Prompt-based memory performs selections based on a similarity metric using vector embeddings and employs a language encoder to obtain vector embeddings for insertions. Both of these are considered non-symbolic executions. In matrix-based memory augmentated LLMs, memory operations are entirely controlled by the neural network, resulting in non-symbolic executions as well.

4. 记忆执行。 这一方面关注记忆操作如何执行，尤其是它们是否符号化。 ChatDB 使用 SQL 对其数据库记忆执行操作，而 SQL 是一种符号语言，因此它本质上是符号化的。 Prompt-based memory 使用向量嵌入基于相似度度量进行选择，并使用语言编码器为插入操作获得向量嵌入。 这两者都被认为是非符号执行。 在用 matrix-based memory 增强的 LLM 中，记忆操作完全由神经网络控制，因此同样产生非符号执行。

5. Interpretability. This aspect refers to the degree of interpretability of memory. In ChatDB, memory is stored in a structured and explicit format, and its operations are symbolic, leading to a high level of interpretability. In the prompt-based memory, interpretability is generally limited due to inherent challenges in interpreting vector embeddings. For matrix-based memory approach, the degree of interpretability is low since memory is completely controlled by the neural network implicitly.

5. 可解释性。 这一方面指记忆的可解释程度。 在 ChatDB 中，记忆以结构化且显式的格式存储，其操作是符号化的，因此具有很高可解释性。 在 prompt-based memory 中，由于解释向量嵌入存在内在困难，可解释性通常有限。 对于 matrix-based memory 方法，由于记忆完全由神经网络隐式控制，因此可解释程度较低。

6. State Tracking. This aspect refers to whether the memory effectively tracks the current state of the LLMs. In the case of ChatDB, its memory accurately tracks the current state of the LLMs. The fruit shop experiment serves as a demonstration, where after processing each record, ChatDB's database memory is updated to reflect the latest state of the fruit shop. This showcases how ChatDB's memory effectively tracks its current state. Thanks to the symbolic memory execution, ChatDB's memory allows for easy rollback to any desired timestamp, providing greater flexibility and controllability. In matrix-based memory approach, memory is continuously updated and altered by the model itself, enabling it to track the current state of the LLMs. However, the prompt-based memory approach simply stores the historical context and only knows what has happened in the past without a clear understanding of the current state.

6. 状态跟踪。 这一方面指记忆是否有效跟踪 LLM 的当前状态。 在 ChatDB 中，它的记忆能够准确跟踪 LLM 的当前状态。 水果店实验提供了一个示范：在处理每条记录后，ChatDB 的数据库记忆都会更新，以反映水果店的最新状态。 这展示了 ChatDB 的记忆如何有效跟踪其当前状态。 得益于符号记忆执行，ChatDB 的记忆可以轻松回滚到任意期望时间戳，从而提供更大的灵活性和可控性。 在 matrix-based memory 方法中，记忆由模型自身持续更新和改变，使其能够跟踪 LLM 的当前状态。 然而，prompt-based memory 方法只是存储历史上下文，只知道过去发生了什么，而无法清楚理解当前状态。

By examining these aspects, we observe the distinctive features and capabilities of ChatDB in comparison to existing approaches. The superiority of ChatDB highlights the advantages of using symbolic memory to augment LLMs.

通过考察这些方面，我们观察到 ChatDB 相较现有方法具有独特特征和能力。 ChatDB 的优势凸显了使用符号记忆增强 LLM 的好处。

4. Evaluation

In this section, we conduct experiments to evaluate the effectiveness of augmenting LLMs with databases as their symbolic memory. Our experiment results demonstrate that ChatDB significantly outperforms the baseline model ChatGPT, highlighting the advantages of symbolic memory integration.

在本节中，我们进行实验，以评估用数据库作为符号记忆增强 LLM 的有效性。 我们的实验结果表明，ChatDB 显著优于基线模型 ChatGPT，突出了符号记忆集成的优势。

4.1. Experimental Setup

As previously mentioned, using a database as symbolic memory is particularly suitable for scenarios that require precise recording and processing of historical information, such as various data management scenarios. To adapt to the use case of ChatDB and enable quantitative comparisons with other models, we constructed a synthetic dataset simulating the management of a fruit shop.

如前所述，使用数据库作为符号记忆特别适合需要精确记录和处理历史信息的场景，例如各种数据管理场景。 为了适配 ChatDB 的用例并支持与其他模型进行定量比较，我们构建了一个模拟水果店管理的合成数据集。

Furthermore, in order to evaluate the performance of the models, we collected a set of 50 questions with annotated standard answers. These questions varied in difficulty, ranging from hard ones that required multi-hop reasoning to easy ones that only required retrieving information from historical data. There are 15 easy questions and 35 hard questions. Each question is independently answered by the models.

此外，为了评估模型性能，我们收集了一组带有标注标准答案的 50 个问题。 这些问题难度不同，从需要多跳推理的困难问题，到只需要从历史数据中检索信息的简单问题都有。 其中有 15 个简单问题和 35 个困难问题。 每个问题都由模型独立回答。

Model Configuration

ChatDB. The LLM used in ChatDB is ChatGPT (GPT-3.5 Turbo), and the hyperparameter temperature is set to $0$. We used the MySQL database as the external symbolic memory. Baseline. We use ChatGPT (GPT-3.5 Turbo) as the baseline model with a maximum token length of 4096. Similar to ChatDB, we set the temperature to $0$.

ChatDB。 ChatDB 中使用的 LLM 是 ChatGPT（GPT-3.5 Turbo），超参数 temperature 设置为 $0$。 我们使用 MySQL 数据库作为外部符号记忆。 基线。 我们使用 ChatGPT（GPT-3.5 Turbo）作为基线模型，最大 token 长度为 4096。 与 ChatDB 类似，我们将 temperature 设置为 $0$。

(a) Purchasing goods: the shop purchases goods from suppliers.

Record: We restocked our store on 2023-01-01 with a new supply of fruits from `ABC' (abc_sup@example.com, 10080). The purchased quantities include 24 kg cherry, at unit prices of 0.8. Our intended selling price of cherry is 1.3 dollars per unit.

ChatDB: Begin to interact with the external database (symbolic memory).

Step1: Insert supplier `ABC' if not exists

INSERT INTO suppliers (supplier_name, contact_number, email)
SELECT 'ABC', '10080', 'abc_sup@example.com'
WHERE NOT EXISTS (SELECT 1 FROM suppliers WHERE supplier_name = 'ABC');

Step2: Insert fruit (set the selling price to NULL and stock quantity to 0) if not exists

INSERT INTO fruits (fruit_name, selling_price, stock_quantity, fruit_type, shelf_life)
SELECT 'cherry', NULL, 0, NULL, NULL
WHERE NOT EXISTS (SELECT 1 FROM fruits WHERE fruit_name = 'cherry');

Step3: Insert purchase

INSERT INTO purchases (supplier_id, purchase_date, total_cost)
VALUES ((SELECT supplier_id FROM suppliers WHERE supplier_name = 'ABC'), '2023-01-01', 24 * 0.8);

Step4: Insert purchase item

INSERT INTO purchase_items (purchase_id, fruit_id, quantity_purchased, cost_per_item, item_total_cost)
VALUES ((SELECT MAX(purchase_id) FROM purchases), (SELECT fruit_id FROM fruits WHERE fruit_name = 'cherry'), 24, 0.8, 24 * 0.8);

Step5: Update the stock quantity of cherry

UPDATE fruits 
SET stock_quantity = stock_quantity + 24
WHERE fruit_name = 'cherry';

Step6: Update the selling price of cherry if given new selling price

UPDATE fruits 
SET selling_price = 1.3
WHERE fruit_name = 'cherry';

(b) Selling goods: customers buy goods from the shop

Record: A sale was made on 2023-01-02 to `Bob Smith' (contact details: 123-456-7893, bob.smith@example.com). The items purchased were 9 kg apple, 4 kg cherry.

ChatDB: Begin to interact with the external database (symbolic memory).

Step1: Insert customer 'Bob Smith' if not exists

INSERT INTO customers (first_name, last_name, phone_number, email)
SELECT 'Bob', 'Smith', '123-456-7893', 'bob.smith@example.com'
WHERE NOT EXISTS (SELECT 1 FROM customers WHERE phone_number = '123-456-7893');

Step2: Insert sale

INSERT INTO sales (customer_id, sale_date, total_price)
VALUES ((SELECT customer_id FROM customers WHERE phone_number = '123-456-7893'), '2023-01-02', (SELECT selling_price FROM fruits WHERE fruit_name = 'apple') * 9 + (SELECT selling_price FROM fruits WHERE fruit_name = 'cherry') * 4);

Step3: Insert sale item

INSERT INTO sale_items (sale_id, fruit_id, quantity_sold, price_per_item, item_total_price)
VALUES ((SELECT MAX(sale_id) FROM sales), (SELECT fruit_id FROM fruits WHERE fruit_name = 'apple'), 9, (SELECT selling_price FROM fruits WHERE fruit_name = 'apple'), (SELECT selling_price FROM fruits WHERE fruit_name = 'apple') * 9),
((SELECT MAX(sale_id) FROM sales), (SELECT fruit_id FROM fruits WHERE fruit_name = 'cherry'), 4, (SELECT selling_price FROM fruits WHERE fruit_name = 'cherry'), (SELECT selling_price FROM fruits WHERE fruit_name = 'cherry') * 4);

Step4: Update the stock quantity of apple and cherry

UPDATE fruits 
SET stock_quantity = CASE 
WHEN fruit_name = 'apple' THEN stock_quantity - 9
WHEN fruit_name = 'cherry' THEN stock_quantity - 4
ELSE stock_quantity
END
WHERE fruit_name IN ('apple', 'cherry');

(c) Return of goods: customers bring back purchased items to the shop.

Record: Because the customer returned their purchase, we are required to undo the sales transaction made by customer `John Doe' (phone: 123-456-7890, email: john.doe@example.com) on 2023-01-05.

ChatDB: Begin to interact with the external database (symbolic memory).

Step1: Find the sale_id for this customer on this date

SELECT sale_id FROM sales 
WHERE customer_id = (SELECT customer_id FROM customers WHERE phone_number = '123-456-7890' AND email = 'john.doe@example.com') AND sale_date = '2023-01-05';

Database response:
+---------+
| sale_id |
+---------+
|    9    |
+---------+

Step2: Get all the fruit_id and quantity_sold for this sale, replace <sale_id> with the results from the previous queries

SELECT fruit_id, quantity_sold FROM sale_items 
WHERE sale_id = 9 [<sale_id>];

Database response:
+----------+---------------+
| fruit_id | quantity_sold |
+----------+---------------+
|    7     |       10      |
|    6     |        9      |
+----------+---------------+

Step3: Increase the stock_quantity for each fruit sold in this sale, replace <quantity_sold> <fruit_id> with the results from the previous queries

UPDATE fruits
SET stock_quantity = stock_quantity + 10 [<quantity_sold>]
WHERE fruit_id = 7 [<fruit_id>];

UPDATE fruits
SET stock_quantity = stock_quantity + 9 [<quantity_sold>]
WHERE fruit_id = 6 [<fruit_id>];

Step4: Delete the sale items for this sale, replace <sale_id> with the results from the previous queries

DELETE FROM sale_items WHERE sale_id = 9 [<sale_id>];

Step5: Delete the sale record, replace <sale_id> with the results from the previous queries

DELETE FROM sales WHERE sale_id = 9 [<sale_id>];

(d) Changing prices: modify the price of the goods.

Record: On 2023-01-05, the sale price of pear in the store was changed to 1.6 dollar per unit.

ChatDB: Begin to interact with the external database (symbolic memory).

Step1: Update the selling price of pear

UPDATE fruits 
SET selling_price = 1.6
WHERE fruit_name = 'pear';

图3：ChatDB 对 Fruit Shop Dataset 记录中四种常见操作的响应示例。“9 [<sale_id>]” 表示在更新后的 SQL 语句中用 9 替换 <sale_id>。由于页面大小限制，省略了一些不必要细节和 ChatDB 的总结。

Dataset

We synthesize a dataset of fruit shop management records, referred to as the "Fruit Shop Dataset". This dataset simulates four common operations in a shop: purchasing, selling, changing prices, and goods returns. We ensure that all historical records are valid and do not encounter issues such as negative inventory. The dataset consists of a total of 73 records, covering one month of records. The dataset is not large and contains only about 13,000 words. Note that ChatDB's performance is generally unaffected by the token length of the dataset. Thus, if ChatDB outperforms ChatGPT when the dataset is small, it indicates that ChatDB is also superior to memory-augmented ChatGPT when the dataset is large.

我们合成了一个水果店管理记录数据集，称为 “Fruit Shop Dataset”。 该数据集模拟商店中的四种常见操作：采购、销售、调价和商品退回。 我们确保所有历史记录都是有效的，并且不会遇到负库存等问题。 该数据集总共包含 73 条记录，覆盖一个月的记录。 该数据集并不大，仅包含约 13,000 个单词。 注意，ChatDB 的性能通常不受数据集 token 长度影响。 因此，如果 ChatDB 在数据集较小时优于 ChatGPT，就表明当数据集较大时，ChatDB 同样优于记忆增强的 ChatGPT。

Processing records

For ChatDB, the first step is to initialize the database. We need to generate a reasonable database schema for the specific task scenario and create tables in the database. The generation of the database schema can be done manually or using LLMs. Next, for each record in the dataset, ChatDB processes them one by one. Using the LLM controller, ChatDB manipulates the external database (i.e., symbolic memory) following Algorithm 1. We provide examples of ChatDB's response to the four common operations in the Fruit Shop Dataset, namely purchasing, selling, changing prices, and goods returns, as shown in Figure 3. It is worth emphasizing that ChatDB processes record one by one, so it is not sensitive to the total number of records. Furthermore, each step of the database operation in ChatDB is symbolic and free from errors. Therefore, in theory, ChatDB can handle an infinite number of historical records without sacrificing performance. However, for ChatGPT or existing memory-augmented LLMs, excessively long historical records can significantly degrade performance. In this experiment, as for the ChatGPT baseline, since the records are not long, we simply treat them as part of the prompt.

对于 ChatDB，第一步是初始化数据库。 我们需要为特定任务场景生成合理的数据库 schema，并在数据库中创建表。 数据库 schema 的生成可以手动完成，也可以使用 LLM 完成。 接下来，对于数据集中的每条记录，ChatDB 会逐条处理。 借助 LLM 控制器，ChatDB 按照算法1操作外部数据库（即符号记忆）。 我们在图3中给出了 ChatDB 对 Fruit Shop Dataset 中四种常见操作的响应示例，即采购、销售、调价和商品退回。 值得强调的是，ChatDB 逐条处理记录，因此它对记录总数并不敏感。 此外，ChatDB 中数据库操作的每一步都是符号化的，并且不会产生错误。 因此，理论上 ChatDB 可以处理无限数量的历史记录而不牺牲性能。 然而，对于 ChatGPT 或现有记忆增强 LLM，过长的历史记录会显著降低性能。 在本实验中，对于 ChatGPT 基线，由于记录并不长，我们简单地将它们作为提示的一部分。

Answering questions

When answering questions, ChatDB no longer requires records to be part of the prompt. After processing the records, the information is stored in symbolic memory. Following Algorithm 1, ChatDB utilizes SQL statements to perform a series of database queries (including calculations) in order to answer the question. On the other hand, ChatGPT includes records as part of the prompt and directly asks the question. The prompt template is shown in Figure 4.

在回答问题时，ChatDB 不再需要把记录作为提示的一部分。 处理记录后，信息被存储在符号记忆中。 按照算法1，ChatDB 利用 SQL 语句执行一系列数据库查询（包括计算），以回答问题。 另一方面，ChatGPT 将记录作为提示的一部分，并直接提出问题。 提示模板如图4所示。

Suppose you are a fruit shop manager and good at analyzing history records.

The fruit shop newly opened on January 1, 2023. Given the history records for the fruit shop in January 2023, which include customer names, transaction dates, fruit prices, quantities purchased, and whether the items were returned, you need to answer some questions.

By default, exclude the sales transactions that have been returned when performing calculations.

Here are the historical records of the fruit shop, which are arranged in chronological order based on the occurrence time, surrounded by triple backticks:

```

{records}

```

Based on the history records, answer the question about the fruit shop:

{question}

图4：ChatGPT 在 Fruit Shop Dataset 中回答问题的提示。占位符 records 和 question 会在实际使用时替换为具体细节。

4.2. Results

The experimental results are presented in Table 2, which clearly demonstrate that ChatDB outperforms ChatGPT with significantly higher accuracy. While ChatGPT is capable of answering easy questions, it falls short in handling hard questions that necessitate multi-hop reasoning and precise calculations. Consequently, ChatGPT exhibits a low accuracy rate for these hard questions. In contrast, ChatDB exhibits a notably high accuracy rate, underscoring the advantages of utilizing a database as symbolic memory. This approach not only prevents error accumulation but also enhances the multi-hop reasoning and precise calculation capabilities of LLMs.

实验结果见表2，清楚表明 ChatDB 以显著更高的准确率优于 ChatGPT。 虽然 ChatGPT 能够回答简单问题，但在处理需要多跳推理和精确计算的困难问题时表现不足。 因此，ChatGPT 在这些困难问题上的准确率较低。 相比之下，ChatDB 展现出明显较高的准确率，凸显了使用数据库作为符号记忆的优势。 该方法不仅防止错误累积，还增强了 LLM 的多跳推理和精确计算能力。

表2：Fruit Shop Dataset 上的问题回答实验结果。总共有 50 个问题，其中 15 个为简单问题，35 个为困难问题。

Model	Easy	Hard	All	Accuracy
ChatGPT	10/15	1/35	11/50	22%
ChatDB (ours)	13/15	28/35	41/50	82%

(a) Question: What was the total revenue for January 2023?

ChatGPT:

To calculate the total revenue for January 2023, we need to sum up the revenue generated from each sales transaction. We can use the following formula to calculate the revenue for each transaction: revenue = quantity * selling price

We also need to exclude the sales transactions that have been returned. Based on the history records, the following sales transactions have been returned:

- Alice Smith on 2023-01-17 - Bob Smith on 2023-01-05 [...]

Therefore, we need to exclude these transactions when calculating the total revenue. Here is the calculation for each sales transaction:

- Bob Smith on 2023-01-02: (9 kg * 3.8 USD/kg) + (4 kg * 1.3 USD/kg) = 39.1 USD [WRONG]: calculation error, should be 39.4

- Sue Davis on 2023-01-02: 1 kg * 3.8 USD/kg = 3.8 USD

- Bob Smith on 2023-01-05: (5 kg * 1.8 USD/kg) + (2 kg * 3.0 USD/kg) + (6 kg * 2.8 USD/kg) = 47.6 USD [WRONG]: calculation error and this transaction has been returned.

[...] [WRONG]: most of the sale transactions are calculated incorrectly.

[STOP]: Because it exceeds the maximum token length.

ChatDB:

Begin to interact with the external database (symbolic memory).

Step1: Calculate the total revenue for January 2023

SELECT SUM(total_price) AS total_revenue
FROM sales
WHERE sale_date >= '2023-01-01' AND sale_date < '2023-02-01';

Database response:
+---------------+
| total_revenue |
+---------------+
|     707.0     |
+---------------+

(b) Question: Which day in January 2023 had the highest revenue?

ChatGPT:

To calculate the revenue for each day, we need to sum up the total sales for that day (excluding any returned items) and multiply by the selling price. We can then compare the revenue for each day to determine which day had the highest revenue.

Here is the breakdown of sales for each day:

- January 2: 9 kg apple at 3.8 = 34.2, 4 kg cherry at 1.3 = 5.2, 3 kg cherry at 1.3 = 3.9, 3 kg orange at 2.1 = 6.3, total sales = 49.6 [WRONG]: should be 43.2

- January 4: 2 kg apple at 1.5 = 3, 4 kg pear at 1.6 = 6.4, 7 kg cherry at 1.3 = 9.1, total sales = 18.5 [WRONG]: should be 27.9

- January 5: 5 kg cherry at 1.9 = 9.5, 10 kg orange at 1.8 = 18, 1 kg cherry at 1.9 = 1.9, 1 kg grape at 3.2 = 3.2, total sales = 32.6 [WRONG]: should be 83

- January 6: 3 kg apple at 1.5 = 4.5, 10 kg grape at 3.2 = 32, total sales = 36.5 [WRONG]: should be 29.4

- January 7: 4 kg pear at 1.6 = 6.4, total sales = 6.4 [WRONG]: should be 37.2

- January 8: 6 kg peach at 3 = 18, total sales = 18 [WRONG]: should be 21.6

[...] [WRONG]: most of the daily revenues are calculated incorrectly.

[STOP]: Because it exceeds the maximum token length.

ChatDB:

Begin to interact with the external database (symbolic memory).

Step1: Calculate the total revenue for each day in January 2023

SELECT sale_date, SUM(total_price) AS revenue
FROM sales
WHERE sale_date BETWEEN '2023-01-01' AND '2023-01-31'
GROUP BY sale_date;

Database response:
+------------+---------+
| sale_date  | revenue |
+------------+---------+
| 2023-01-02 |   43.2  |
| 2023-01-04 |   27.9  |
|     ...    |    ...  |
+----------------------+

Step2: Find the day with the highest revenue

SELECT sale_date, SUM(total_price) AS revenue
FROM sales
WHERE sale_date BETWEEN '2023-01-01' AND '2023-01-31'
GROUP BY sale_date ORDER BY revenue DESC LIMIT 1;

Database response:
+------------+---------+
| sale_date  | revenue |
+------------+---------+
| 2023-01-30 |  119.7  |
+------------+---------+

(c) Question: What was the average weight of fruit per sales transaction in January 2023?

ChatGPT: To calculate the average weight of fruit per sales transaction, we need to first identify all the sales transactions in January 2023 and calculate the total weight of fruit sold in those transactions. Then, we can divide the total weight by the number of sales transactions to get the average weight per transaction.

Here are the sales transactions in January 2023:

- Bob Smith on 2023-01-02: 9 kg apple, 4 kg cherry

- Sue Davis on 2023-01-02: 1 kg apple

- Bob Smith on 2023-01-05: 5 kg orange, 2 kg peach, 6 kg pear [WRONG]: this transaction has been returned.

- Sue Davis on 2023-01-05: 5 kg pear, 10 kg orange, 1 kg cherry

- Alice Smith on 2023-01-05: 5 kg cherry

[...] [WRONG]: not find out all valid sale transactions.

To calculate the total weight of fruit sold, we can add up the quantities of each fruit sold in the above transactions:

- Apples: 16 kg - Cherries: 12 kg - Oranges: 10 kg - Pears: 18 kg - Kiwis: 12 kg - Grapes: 18 kg - Bananas: 27 kg - Strawberries: 7 kg - Peaches: 9 kg - Plums: 8 kg

The total weight of fruit sold in January 2023 is 127 kg. [WRONG]: It should be 278. Even according to the calculation above, the total should be 137, not 127.

To calculate the average weight per sales transaction, we need to divide the total weight by the number of sales transactions. There are 18 sales transactions in January 2023, [WRONG]: It should be 28 valid sale transactions in total. so the average weight [STOP]: Because it exceeds the maximum token length.

ChatDB:

Begin to interact with the external database (symbolic memory).

Step1: Calculate the total weight of fruit sold in January 2023

SELECT SUM(si.quantity_sold) as total_weight
FROM sale_items si
JOIN sales s ON si.sale_id = s.sale_id
WHERE s.sale_date BETWEEN '2023-01-01'
AND '2023-01-31';

Database response:
+--------------+
| total_weight |
+--------------+
|     278      |
+--------------+

Step2: Count the number of sales transactions in January 2023

SELECT COUNT(DISTINCT s.sale_id) as num_sales
FROM sales s
WHERE s.sale_date BETWEEN '2023-01-01'
AND '2023-01-31';

Database response:
+-----------+
| num_sales |
+-----------+
|     28    |
+-----------+

Step3: Calculate the average weight of fruit per sales transaction in January 2023, replace <total_weight> and <num_sales> with the results from the previous queries

SELECT 278 [<total_weight>] / 28
[<num_sales>]
as avg_weight_per_sale;

Database response:
+---------------------+
| avg_weight_per_sale |
+---------------------+
|         9.93        |
+---------------------+

图5：ChatDB 和 ChatGPT 回答问题的示例。ChatGPT 未能正确回答任何问题，而 ChatDB 成功回答全部问题。ChatGPT 的问题用红色突出显示。为简洁起见，省略了一些无关细节和 ChatDB 的总结。灰色省略号 [...] 表示响应被裁剪。

We present several examples of the two models answering questions in Figure 5 for comparison. In all these examples, ChatDB correctly answers the questions while ChatGPT fails. ChatGPT often exhibits errors in calculating the total price of each sale transaction, as observed in Figure 5(a). Sometimes, the formulas are correct but the calculations are wrong, while other times, even the formulas are incorrect. In addition, ChatGPT struggles to find all valid sale transactions, leading to errors in its answering process. This issue is commonly encountered and evident in all these examples. Furthermore, ChatGPT tends to make sequential errors, resulting in significant error accumulation.

我们在图5中展示了两个模型回答问题的若干示例，以进行比较。 在所有这些示例中，ChatDB 都正确回答问题，而 ChatGPT 失败。 如图5(a) 所示，ChatGPT 经常在计算每笔销售交易总价时出错。 有时公式是正确的但计算错误，而另一些时候甚至公式也是错误的。 此外，ChatGPT 难以找出所有有效销售交易，从而在回答过程中产生错误。 这个问题很常见，并且在所有这些示例中都很明显。 此外，ChatGPT 倾向于产生连续错误，导致显著的错误累积。

In contrast, ChatDB performs quite well in these examples. During the initial processing of records, symbolic operations (i.e., SQL operations) are applied to manipulate the database (i.e., symbolic memory), ensuring that all information is stored in a structured form within the database. When answering questions, ChatDB generates SQL statements to query the database. The three examples demonstrate the effectiveness of ChatDB in solving problems requiring one, two, and three chain-of-memory steps, respectively. We can observe that ChatDB accurately answers the questions, and the execution logic of the chain-of-memory is clear, with each step tightly connected and approaching the ultimate answer. From these examples, the advantages of ChatDB are evident in two aspects:

1. Through the chain-of-memory approach, complex problems are decomposed into multiple steps of memory operations, simplifying the problem's complexity. Each step's result is accurately stored as an intermediate outcome and used in subsequent steps, which greatly assists in complex reasoning.

2. Symbolic memory enables precise operations and calculations. ChatDB delegates many calculational tasks to the external database by executing SQL statements, ensuring the accuracy of each step and preventing error accumulation.

In summary, by leveraging external databases as symbolic memory, ChatDB significantly outperforms ChatGPT in this experiment.

相比之下，ChatDB 在这些示例中表现相当好。 在初始处理记录时，符号操作（即 SQL 操作）被用于操作数据库（即符号记忆），确保所有信息都以结构化形式存储在数据库中。 在回答问题时，ChatDB 生成 SQL 语句来查询数据库。 这三个示例分别展示了 ChatDB 在解决需要一、二、三步 chain-of-memory 的问题时的有效性。 我们可以看到，ChatDB 准确回答了问题，而且 chain-of-memory 的执行逻辑清晰，每一步紧密连接并逐步接近最终答案。 从这些示例可以看出，ChatDB 的优势体现在两个方面：

1. 通过 chain-of-memory 方法，复杂问题被分解为多个记忆操作步骤，从而简化问题复杂度。每一步的结果都被准确存储为中间结果，并用于后续步骤，这极大帮助了复杂推理。

2. 符号记忆支持精确操作和计算。ChatDB 通过执行 SQL 语句，将许多计算任务交给外部数据库，确保每一步的准确性并防止错误累积。

总之，通过利用外部数据库作为符号记忆，ChatDB 在本实验中显著优于 ChatGPT。

5. Conclusion

In this paper, we introduce ChatDB, a framework that augments LLMs with symbolic memory in the form of databases. We demonstrate the advantages and capabilities of symbolic memory and chain-of-memory approach in enhancing complex reasoning and preventing error accumulation. By providing a precise storage mechanism for intermediate results, symbolic memory enables accurate and reliable operations. Moreover, the use of symbolic languages, such as SQL, allows symbolic computation and manipulation of stored information. Through experimental evaluation, we observe a significant improvement in performance with ChatDB compared to ChatGPT. The integration of symbolic memory in ChatDB substantially enhances the model's ability to handle various queries and reasoning tasks in management settings. This improvement highlights the benefits and effectiveness of leveraging symbolic memory in LLMs.

在本文中，我们介绍了 ChatDB，这是一个以数据库形式用符号记忆增强 LLM 的框架。 我们展示了符号记忆和 chain-of-memory 方法在增强复杂推理并防止错误累积方面的优势和能力。 通过为中间结果提供精确存储机制，符号记忆支持准确可靠的操作。 此外，使用 SQL 等符号语言，可以对存储信息进行符号计算和操作。 通过实验评估，我们观察到 ChatDB 相较 ChatGPT 在性能上有显著提升。 在 ChatDB 中集成符号记忆，显著增强了模型在管理场景中处理各种查询和推理任务的能力。 这一提升突出了在 LLM 中利用符号记忆的收益和有效性。