---

AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents

MemoryAgentKnowledge Graph160+120+IJCAI 2025CCF-A莫斯科人工智能研究院斯科尔科大学伦敦数学科学研究所牛津大学

Anokhin P, Semenov N, Sorokin A, et al. AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents. IJCAI 2025.

https://github.com/AIRI-Institute/AriGraph

---

AriGraph：用情景记忆为 LLM 智能体学习知识图谱世界模型

Abstract

Advancements in the capabilities of Large Language Models (LLMs) have created a promising foundation for developing autonomous agents. With the right tools, these agents could learn to solve tasks in new environments by accumulating and updating their knowledge. Current LLM-based agents process past experiences using a full history of observations, summarization, retrieval augmentation. However, these unstructured memory representations do not facilitate the reasoning and planning essential for complex decision-making. In our study, we introduce AriGraph, a novel method wherein the agent constructs and updates a memory graph that integrates semantic and episodic memories while exploring the environment. We demonstrate that our Ariadne LLM agent, consisting of the proposed memory architecture augmented with planning and decision-making, effectively handles complex tasks within interactive text game environments difficult even for human players. Results show that our approach markedly outperforms other established memory methods and strong RL baselines in a range of problems of varying complexity. Additionally, AriGraph demonstrates competitive performance compared to dedicated knowledge graph-based methods in static multi-hop question-answering.

大语言模型（LLM）能力的进步，为开发自主智能体奠定了很有前景的基础。 借助合适的工具，这些智能体可以通过积累并更新知识，学会在新环境中解决任务。 当前基于 LLM 的智能体使用完整观察历史、总结和检索增强来处理过去经验。 然而，这些非结构化记忆表示并不利于复杂决策所必需的推理与规划。 在我们的研究中，我们提出 AriGraph，这是一种新方法：智能体在探索环境时构建并更新一个整合语义记忆和情景记忆的记忆图。 我们证明，由所提出的记忆架构并结合规划和决策构成的 Ariadne LLM 智能体，能够有效处理交互式文本游戏环境中的复杂任务，而这些任务甚至对人类玩家也很困难。 结果显示，我们的方法在一系列复杂度不同的问题上显著优于其他成熟记忆方法和强 RL 基线。 此外，AriGraph 在静态多跳问答中也展现出相对于专用知识图谱方法的竞争性表现。

1. Introduction

Impressive language generation capabilities of large language models (LLMs) has sparked substantial interest in their application as core components for creating autonomous agents capable of interacting with dynamic environments and executing complex tasks. Over the past year, the research community has explored general architectures and core modules for such LLM agents. A crucial property of a general cognitive agent is its ability to accumulate and use knowledge. A long-term memory allows an agent to store and recall past experiences and knowledge, enabling it to learn from previous encounters and make informed decisions. However, the question of the best way to equip an agent with these capabilities remains open. Despite the constraints inherent in transformer architectures, contemporary methods enable LLMs to manage contexts encompassing millions of tokens. However, this approach proves inefficient for agents required to maintain continuous interaction with their environment. Such agents must hold an entire historical context in memory to perform actions, which is not only costly but also limited in handling complex logic hidden in vast amounts of information. Research into alternative frameworks like Recurrent Memory Transformer and MAMBA seeks to provide long-term memory solutions, though these models are still in their infancy.

大语言模型（LLM）令人印象深刻的语言生成能力，激发了人们将其作为核心组件来创建自主智能体的浓厚兴趣；这些智能体能够与动态环境交互并执行复杂任务。 过去一年，研究社区已经探索了这类 LLM 智能体的通用架构和核心模块。 通用认知智能体的一项关键属性，是积累和使用知识的能力。 长期记忆允许智能体存储并回忆过去经验和知识，使其能够从以往遭遇中学习并做出知情决策。 然而，如何最好地为智能体配备这些能力仍是一个开放问题。 尽管 transformer 架构存在固有限制，当前方法已经使 LLM 能够管理包含数百万 token 的上下文。 然而，对于需要与环境持续交互的智能体而言，这种方法效率很低。 这类智能体必须在记忆中保留完整历史上下文才能执行动作，这不仅成本高，而且在处理隐藏于海量信息中的复杂逻辑时能力有限。 围绕 Recurrent Memory Transformer 和 MAMBA 等替代框架的研究试图提供长期记忆解决方案，但这些模型仍处于早期阶段。

图1：Ariadne 智能体和 AriGraph 记忆概览。AriGraph 将语义知识图谱与过去经验结合，显著提升 LLM 智能体在文本游戏中的表现。

Currently, the most popular solution for incorporating memory to LLM agents is the Retrieval-Augmented Generation (RAG) approach. RAG in a form of vector retrieval leverages an external database to enhance the model's prompt with relevant information. This technique is commonly used in memory architectures for LLM agents, often to recall specific observations or learned skills. However, it suffers from unstructured nature, greatly reducing the ability to retrieve related information, which may be scattered throughout the agent's memory. These limitations can be overcome by using knowledge graphs as database. This approach has also experienced a resurgence with the advent of LLMs. However, for a robust memory architecture, integrating both structured and unstructured data is essential. In cognitive science, this integration parallels the concepts of semantic and episodic memories. Semantic memory encompasses factual knowledge about the world, whereas episodic memory pertains to personal experiences, which often contain richer and more detailed information. Though traditionally considered separate due to their distinct neurological representations, recent studies suggest these memory types are interconnected. Semantic knowledge is built upon the foundation of episodic memory and subsequently provides a structured base for associative memory. This allows for the integration of various memory aspects, including episodic memories themselves.

目前，为 LLM 智能体引入记忆最流行的方案是检索增强生成（RAG）。 向量检索形式的 RAG 利用外部数据库，用相关信息增强模型提示。 这种技术常用于 LLM 智能体的记忆架构中，通常用于回忆特定观察或已学会的技能。 然而，它受非结构化特性限制，极大削弱了检索相关信息的能力，而这些信息可能分散在智能体的记忆各处。 使用知识图谱作为数据库可以克服这些限制。 随着 LLM 的出现，这一路线也重新兴起。 然而，对于一个强健的记忆架构而言，同时整合结构化和非结构化数据是必要的。 在认知科学中，这种整合对应于语义记忆和情景记忆的概念。 语义记忆包含关于世界的事实知识，而情景记忆涉及个人经验，这些经验通常包含更丰富、更详细的信息。 尽管由于神经表征不同，二者传统上被认为是分离的，但近期研究表明这些记忆类型是相互连接的。 语义知识建立在情景记忆的基础之上，随后为联想记忆提供结构化基础。 这使得整合多种记忆方面成为可能，其中也包括情景记忆本身。

In our research, we have developed a memory architecture called Ariadne's Graph (AriGraph), that integrates semantic and episodic memories within a memory graph framework. A knowledge graph represents a network of interconnected semantic knowledge, while episodic memories are depicted as episodic edges that can connect multiple relations within the graph. As an agent interacts with environment, it learns joint semantic and episodic world model by updating and extending knowledge graph based memory. This architecture not only serves as a foundational memory framework but also aids in environmental modeling, improving spatial orientation and exploration capabilities. For the general framework of our LLM agent called Ariadne, we employed pipeline of memory retrieval, planing and decision making. For evaluation of proposed methods we set up experiments to study two research questions.

在我们的研究中，我们开发了一种名为 Ariadne's Graph（AriGraph）的记忆架构，它在记忆图框架内整合语义记忆和情景记忆。 知识图谱表示相互连接的语义知识网络，而情景记忆则表示为情景边，可以连接图中的多种关系。 当智能体与环境交互时，它通过更新并扩展基于知识图谱的记忆，学习联合的语义与情景世界模型。 该架构不仅作为基础记忆框架，也有助于环境建模，从而提升空间定向和探索能力。 对于我们名为 Ariadne 的 LLM 智能体通用框架，我们采用了记忆检索、规划和决策流水线。 为了评估所提出的方法，我们设计实验研究两个问题。

• RQ1. Can LLM based agents learn useful structured world model from scratch via interaction with an environment?
• RQ2. Does structured knowledge representation improve retrieval of relevant facts from memory and enable effective exploration?

• RQ1. 基于 LLM 的智能体能否通过与环境交互从零开始学习有用的结构化世界模型？
• RQ2. 结构化知识表示是否能改进从记忆中检索相关事实的能力，并支持有效探索？

We evaluated our agent in complex interactive tasks in Textworld and NetHack environments. Experimental results demonstrate that our agent Ariadne can effectively learn through interactions with environment and significantly outperforms other memory approaches for LLMs such as full history, summarization, RAG, Simulacra and Reflexion. We also show that our method outperforms existing reinforcement learning (RL) baselines. We also evaluated our approach on the classical roguelike game NetHack, where our agent with local observations achieved scores comparable to an agent with ground-truth knowledge. Although AriGraph was originally designed for an agent interacting with the environment, it also demonstrates competitive performance on multi-hop question answering tasks.

我们在 TextWorld 和 NetHack 环境中的复杂交互任务上评估了智能体。 实验结果表明，我们的智能体 Ariadne 能够通过与环境交互有效学习，并显著优于 LLM 的其他记忆方法，例如完整历史、总结、RAG、Simulacra 和 Reflexion。 我们还表明，我们的方法优于现有强化学习（RL）基线。 我们也在经典 roguelike 游戏 NetHack 上评估了我们的方法；在该环境中，使用局部观察的智能体取得了与拥有真实知识的智能体相当的分数。 尽管 AriGraph 最初是为与环境交互的智能体设计的，但它在多跳问答任务上也展现出有竞争力的表现。

2. AriGraph World Model

Memory graph structure. AriGraph world model $G=(V_s,E_s,V_e,E_e)$ consists of semantic $(V_s,E_s)$ and episodic memory $(V_e,E_e)$ vertices and edges (see Figure 2). At each step $t$ agent receives observation $o_t$ and sends action $a_t$ back to the environment. The environment also returns rewards $r_t$ that are not visible to the LLM agent but are used to evaluate its performance. The agent continuously learns world model $G$ by extracting semantic triplets $(objec{t}_1,relation,objec{t}_2)$ from textual observations $o_t$.

记忆图结构。 AriGraph 世界模型 $G=(V_s,E_s,V_e,E_e)$ 由语义 $(V_s,E_s)$ 与情景记忆 $(V_e,E_e)$ 的顶点和边组成（见 图2）。 在每一步 $t$，智能体接收观察 $o_t$，并向环境返回动作 $a_t$。 环境还会返回奖励 $r_t$；这些奖励对 LLM 智能体不可见，但用于评估其性能。 智能体通过从文本观察 $o_t$ 中抽取语义三元组 $(objec{t}_1,relation,objec{t}_2)$，持续学习世界模型 $G$。

• $V_s$ is a set of semantic vertices. Semantic vertices correspond to objects extracted from triplets.
• $E_s$ is a set of semantic edges. Semantic edge is a tuple $(v,rel,u)$, where $u$, $v$ are semantic vertices and $rel$ is a relationship between them. Semantic edges essentially represent triplets integrated in the semantic memory.
• $V_e$ is a set of episodic vertices. Each episodic vertex corresponds to an observation received from the environment at the respective step $v_e^t=o_t$.
• $E_e$ is a set of episodic edges. Each episodic edge $e_e^t=(v_e^t,E_s^t)$ connects all semantic triplets $E_s^t$ extracted from $o_t$ with each other and corresponding episodic vertex $v_e^t$. In other words episodic edges represent temporal relationship "happened at the same time".

• $V_s$ 是语义顶点集合。语义顶点对应于从三元组中抽取出的对象。
• $E_s$ 是语义边集合。语义边是一个元组 $(v,rel,u)$，其中 $u$ 和 $v$ 是语义顶点，$rel$ 是它们之间的关系。语义边本质上表示整合到语义记忆中的三元组。
• $V_e$ 是情景顶点集合。每个情景顶点对应于在相应步骤从环境收到的一条观察 $v_e^t=o_t$。
• $E_e$ 是情景边集合。每条情景边 $e_e^t=(v_e^t,E_s^t)$ 将从 $o_t$ 中抽取的所有语义三元组 $E_s^t$ 彼此连接，并连接到对应的情景顶点 $v_e^t$。换言之，情景边表示“同时发生”的时间关系。

图2：AriGraph 世界模型与 Ariadne 认知架构。左：交互过程中学习的语义知识和情景知识。右：Ariadne 使用 AriGraph 进行探索、规划与决策。

Constructing AriGraph. Interaction with the environment can provide the agent with an information about the world to create new or update previously acquired knowledge. Given new observation $o_t$, LLM agent extracts new triplets as semantic vertices $V_s^t$ and edges $E_s^t$. To find already existing knowledge about the objects mentioned in $o_t$ a set of all semantic edges $E_s^{rel}$ incident to vertices $V_s^t$ is filtered out. Then outdated edges in $E_s^{rel}$ are detected by comparing them with $E_s^t$ and removed from the graph. After clearing outdated knowledge we expand semantic memory with $V_s^t$ and $E_s^t$. Episodic memory is updated by simply adding new episodic vertex $v_e^t$ containing $o_t$ and new episodic edge that connect all edges in $E_s^t$ with $v_e^t$. Episodic nodes store agent's past history and episodic edges connects all knowledge received at the same step. See Appendix for prompts used to extract new triplets and detect outdated knowledge.

构建 AriGraph。 与环境交互可以为智能体提供关于世界的信息，使其创建新知识或更新先前获得的知识。 给定新的观察 $o_t$，LLM 智能体抽取新的三元组作为语义顶点 $V_s^t$ 和语义边 $E_s^t$。 为了找到关于 $o_t$ 中提到对象的已有知识，我们会筛出与顶点 $V_s^t$ 相连的所有语义边集合 $E_s^{rel}$。 然后，通过将 $E_s^{rel}$ 与 $E_s^t$ 比较，检测其中的过时边并从图中移除。 清除过时知识后，我们用 $V_s^t$ 和 $E_s^t$ 扩展语义记忆。 情景记忆的更新很简单：加入包含 $o_t$ 的新情景顶点 $v_e^t$，以及把 $E_s^t$ 中所有边与 $v_e^t$ 连接起来的新情景边。 情景节点存储智能体的过去历史，情景边则连接同一步获得的所有知识。 抽取新三元组和检测过时知识所用的提示见附录。

Retrieval from AriGraph. For successful decision-making in a partially observable environment, the agent needs to be able to retrieve relevant knowledge. Retrieval from the AriGraph memory consists of two procedures: (1) a semantic search returns the most relevant triplets (semantic edges) and (2) an episodic search that, given extracted triplets, returns the most relevant episodic vertices $V_e$. The pseudo-code for the search is presented in the Algorithm 1.

从 AriGraph 检索。 为了在部分可观察环境中成功决策，智能体需要能够检索相关知识。 从 AriGraph 记忆中检索包含两个过程：（1）语义搜索返回最相关的三元组（语义边）；（2）情景搜索在给定抽取出的三元组后，返回最相关的情景顶点 $V_e$。 搜索的伪代码见 算法1。

Algorithm 1: Memory Graph Search

Input: set of queries $Q$, $V_s$, $E_s$, $V_e$, $E_e$, number of episodic vertices $k$, semantic search depth $d$ and width $w$
Result: retrieved episodic vertices $V_e^Q$, retrieved semantic triplets $E_s^Q$

1. $E_s^Q\leftarrow \mathrm{\varnothing }$
2. for each $q$ in $Q$ do
3. $E_s^{\prime }\leftarrow \mathtt{\text{SemanticSearch}}(q,V_s,E_s,d,w)$
4. $E_s^Q\leftarrow E_s^Q\cup E_s^{\prime }$
5. end for
6. $V_e^Q\leftarrow \mathtt{\text{EpisodicSearch}}(E_s^Q,V_e,E_e,k)$
7. return $E_s^Q$, $V_e^Q$

Semantic search relies on semantic similarity and semantic graph structure to recall the most relevant triplets. Given a query, the retriever (pre-trained Contriever model) selects the most relevant semantic triplets. Then, the set of vertices incident to the found triplets is used to recursively retrieve new edges from the graph. Depth and breadth of the search can be controlled by respective hyperparameters $d$ and $w$. For details see Appendix.

语义搜索依赖语义相似度和语义图结构来回忆最相关的三元组。 给定一个查询，检索器（预训练 Contriever 模型）会选择最相关的语义三元组。 随后，与找到的三元组相连的顶点集合被用于从图中递归检索新的边。 搜索深度和宽度可以分别由超参数 $d$ 和 $w$ 控制。 细节见附录。

Episodic search starts with the results of the semantic search as an input. Episodic edges link the input triplets with past episodic vertices representing observations. The number of input triplets associated with a particular episodic vertex is used to calculate their relevance:

情景搜索以语义搜索的结果作为输入。 情景边将输入三元组与表示观察的过去情景顶点连接起来。 与某个特定情景顶点关联的输入三元组数量被用于计算其相关性：

$$\begin{array}{rl}rel(v_e^i)& =\frac{n_i}{max(N_i,1)}\log (max(N_i,1)).\end{array}$$

where $n_i$ is a number of input triplets incident to episodic edge $e^i$, $N_i$ is a total number triplets (semantic edges) incident to $e^i$ and $\log (max(N_i,1))$ is a weighting factor to prevent high scores for low information observations. We apply a $lo{g}_2(N_i)$ scaling to give more weight to observations with more extracted triplets. Additionally, observations containing exactly one triplet are assigned zero weight, as they are unlikely to provide information beyond the triplet itself. $k$ most relevant episodic vertices (containing respective observations) are returned as a result of the episodic search.

其中 $n_i$ 是与情景边 $e^i$ 相连的输入三元组数量，$N_i$ 是与 $e^i$ 相连的三元组（语义边）总数，而 $\log (max(N_i,1))$ 是一个权重因子，用于防止低信息量观察获得高分。 我们应用 $lo{g}_2(N_i)$ 缩放，让包含更多抽取三元组的观察获得更高权重。 此外，恰好只包含一个三元组的观察被赋予零权重，因为它们不太可能提供超出该三元组本身的信息。 情景搜索最终返回 $k$ 个最相关的情景顶点（包含相应观察）。

3. Ariadne cognitive architecture

To test utility of AriGraph world modelling method we propose an agentic architecture called Ariadne. Ariadne agent interacts with an unknown environment to accomplish a goal set by a user. Throughout this process, at each time step, the agent learns a world model, plans and executes actions. Ariadne has long-term memory stored as AriGraph and working memory containing information for current planning and decision making.

为了测试 AriGraph 世界建模方法的效用，我们提出了一种名为 Ariadne 的智能体架构。 Ariadne 智能体与未知环境交互，以完成用户设定的目标。 在整个过程中，智能体在每个时间步学习世界模型、规划并执行动作。 Ariadne 拥有以 AriGraph 形式存储的长期记忆，以及包含当前规划和决策所需信息的工作记忆。

Given an observation the agent updates world model and retrieves semantic and episodic knowledge from AriGraph to working memory. Working memory is also populated with a final goal description, current observation, history of recent observation and actions. At the planning stage, Ariadne agent uses content of working memory to create new or update existing plan as a series of task-relevant sub-goals, each accompanied by a concise description. The planning module also evaluates the outcomes of actions based on feedback from the environment after each action at step $t-1$, adjusting the plan accordingly.

给定一条观察，智能体会更新世界模型，并从 AriGraph 中检索语义知识和情景知识到工作记忆中。 工作记忆还会填入最终目标描述、当前观察、近期观察和动作历史。 在规划阶段，Ariadne 智能体使用工作记忆内容来创建新计划或更新现有计划，将其表示为一系列与任务相关的子目标，每个子目标都配有简洁描述。 规划模块还会根据步骤 $t-1$ 中每个动作之后来自环境的反馈评估动作结果，并相应调整计划。

The revised plan is added to the working memory which is accessed by the decision-making module, tasked with selecting the most suitable action aligned with the current plan’s objectives. This module adheres to the ReAct framework, requiring the agent to articulate the rationale behind an action before execution. Separation of planning from decision-making enables LLMs to focus on distinct cognitive processes. In text-based environments an agent selects an action from the list of valid actions. Our agent can also use graph specific function for navigation utilizing its memory module. It extends its action space with "go to location" type commands and infers an optimal route to a target location using spatial relations stored in a semantic graph.

修订后的计划会加入工作记忆，决策模块会访问该工作记忆，并负责选择与当前计划目标一致的最合适动作。 该模块遵循 ReAct 框架，要求智能体在执行动作之前阐明动作背后的理由。 将规划与决策分离，使 LLM 能够专注于不同的认知过程。 在基于文本的环境中，智能体从有效动作列表中选择动作。 我们的智能体也可以利用其记忆模块，使用图特定函数进行导航。 它用“go to location”类型命令扩展动作空间，并使用存储在语义图中的空间关系推断到目标位置的最优路线。

4. Experimental Setup

4.1 TextWorld interactive environments

图3：AriGraph 世界模型使 Ariadne 智能体能够成功解决多种文本游戏。归一化分数相对于每个环境可获得的最大分数计算，误差条表示标准差。

We compared Ariadne agent with alternative methods in a series of text based games involving spatial navigation, object collection and tool manipulation. Detailed descriptions of each game, including their difficulty levels and environmental maps, can be found in the Appendix. All these games can be considered Partially Observable MDPs (POMDPs). Such games have long been benchmarks for researching agents capable of effectively remembering information and establishing long-term dependencies.

我们在一系列涉及空间导航、物品收集和工具操作的文本游戏中，将 Ariadne 智能体与其他方法进行了比较。 每个游戏的详细描述，包括难度等级和环境地图，见附录。 所有这些游戏都可以被视为部分可观察 MDP（POMDP）。 这类游戏长期以来一直是研究能够有效记忆信息并建立长期依赖关系的智能体的基准。

Treasure Hunting. The primary objective is to retrieve the hidden treasure, with a series of rooms providing keys and clues leading to the final goal. The basic variation has 12 rooms and 4 keys, hard one has 16 rooms and 5 keys and hardest contains 36 rooms, 7 keys and additional distract items in every room.

寻宝。 主要目标是取回隐藏宝藏，一系列房间会提供通向最终目标的钥匙和线索。 基础版本有 12 个房间和 4 把钥匙，困难版本有 16 个房间和 5 把钥匙，最困难版本包含 36 个房间、7 把钥匙，并在每个房间都有额外干扰物品。

Cleaning. The goal is to clean a house by identifying and returning misplaced items to their correct locations. Environment consists of 9 rooms (kitchen, pool, etc.) and contains 11 misplaced items (among many other items). To solve the problem, the agent needs to memorize the location of rooms and objects, as well as reason about objects placement.

清洁。 目标是通过识别错放物品并将其归还到正确位置来清理房屋。 环境由 9 个房间（厨房、泳池等）组成，并包含 11 个错放物品（以及许多其他物品）。 为了解决该问题，智能体需要记住房间和物品的位置，并推理物品应放置的位置。

Cooking. The goal is to prepare and consume a meal by following a recipe, selecting the correct ingredients, and using appropriate tools, while navigating in multi-room house. The task is testing agents ability to remember relevant information and plan according to it. Basic difficulty task features 9 locations and 3 ingredients and hard task features 12 locations and 4 ingredients, while hardest task also features closed doors and inventory management.

烹饪。 目标是在多房间房屋中导航的同时，按照食谱准备并食用一餐，选择正确食材并使用合适工具。 该任务测试智能体记住相关信息并据此规划的能力。 基础难度任务包含 9 个地点和 3 种食材，困难任务包含 12 个地点和 4 种食材，而最困难任务还包含上锁的门和库存管理。

For baselines we used Ariadne's planning and decision making module with one of the following types of memory instead of AriGraph model: full history of observations and actions, iterative summarization, RAG, RAG with Reflexion, and Simulacra - memory implementation from Generative Agents.

对于基线，我们使用 Ariadne 的规划和决策模块，但用以下记忆类型之一替代 AriGraph 模型：观察和动作的完整历史、迭代总结、RAG、带 Reflexion 的 RAG，以及来自 Generative Agents 的记忆实现 Simulacra。

Full history involves retaining a complete record of all observations and actions to inform decision-making at every step. Summarization, as an alternative to storing the full history, focuses on retaining only the necessary information while discarding the rest. The standard RAG baseline retrieves top-k memories based on their similarity score to the current observation and plan. Simulacra features a scoring mechanism that integrates recency, importance, and relevance, alongside reflections on the extracted memories. The Reflexion baseline differs from other methods in its approach, as it operates over multiple trials. After failing a trial, the agent reflects on its trajectories to document information that may assist in solving the task in subsequent trials. We used the gpt-4-0125-preview as LLM backbone for AriGraph and other LLM-based baselines.

完整历史会保留所有观察和动作的完整记录，用于每一步决策。 总结作为存储完整历史的替代方案，专注于只保留必要信息并丢弃其余信息。 标准 RAG 基线根据当前观察和计划的相似度分数检索 top-k 记忆。 Simulacra 具有一种评分机制，将近因性、重要性和相关性结合起来，并加入对抽取记忆的反思。 Reflexion 基线的方法与其他方法不同，因为它在多次试验上运行。 一次试验失败后，智能体会反思其轨迹，记录可能有助于后续试验解决任务的信息。 我们使用 gpt-4-0125-preview 作为 AriGraph 和其他基于 LLM 的基线的 LLM backbone。

Additionally, we tested our architecture on a variation of the cooking test from Adhikari et al. to compare it with RL baselines. These tasks have 4 levels of difficulty, however, they are significantly simpler than our main tasks, having fewer locations, ingredients, and required actions (Appendix). For RL baselines, we collect the best results reported by Adhikari et al., Tuli et al., and Basu et al. for the GATA, LTL-GATA, and EXPLORER architectures on the Cooking task with four difficulties levels from Adhikari et al. To estimate human performance in the same games, we developed a graphical user interface, allowing volunteers to play basic versions of the Treasure Hunt, The Cleaning, and the Cooking. After collecting the data, we excluded sessions where the game was not completed.

此外，为了与 RL 基线比较，我们在 Adhikari 等人提出的烹饪测试变体上测试了我们的架构。 这些任务有 4 个难度等级，但它们明显比我们的主要任务更简单，地点、食材和所需动作都更少（见附录）。 对于 RL 基线，我们收集了 Adhikari 等人、Tuli 等人和 Basu 等人报告的最佳结果，涵盖 Adhikari 等人四个难度等级烹饪任务上的 GATA、LTL-GATA 和 EXPLORER 架构。 为了估计同一批游戏中的人类表现，我们开发了一个图形用户界面，让志愿者可以玩基础版本的 Treasure Hunt、The Cleaning 和 Cooking。 收集数据后，我们排除了未完成游戏的会话。

4.2 NetHack environment

NetHack is a classic roguelike adventure game featuring procedurally generated multi-level dungeon (see Appendix for a dungeon level example). It poses significant challenges for both LLM-based and RL-based approaches, requiring complex exploration, resource management, and strategic planning. We based our experiments on NetPlay agent, which demonstrates state-of-the-art performance among LLM agents that do not rely on finetuning or RL. In NetPlay agent receives textual observations containing all information about current explored dungeon level. These observations (Level obs) effectively function as handcrafted memory oracle for the agent.

NetHack 是一款经典 roguelike 冒险游戏，具有程序生成的多层地牢（地牢层级示例见附录）。 它对基于 LLM 和基于 RL 的方法都提出了显著挑战，需要复杂探索、资源管理和战略规划。 我们的实验基于 NetPlay 智能体；在不依赖微调或 RL 的 LLM 智能体中，该智能体展现出最先进的表现。 在 NetPlay 中，智能体接收包含当前已探索地牢层级所有信息的文本观察。 这些观察（Level obs）实际上充当了智能体的手工记忆 oracle。

To evaluate our Ariadne agent, we restricted textual observations to agent's current room or corridor (Room Obs), testing whether AriGraph world model could compensate for this restriction by remembering all relevant level information. We compare three agents. The first is NetPlay [Room obs] with restricted textual observations, the second is our Ariadne [Room obs] agent that receives Room Obs and updates AriGraph, and the last is NetPlay [Level obs] with access to inforation about explored level.

为了评估我们的 Ariadne 智能体，我们将文本观察限制为智能体当前所在房间或走廊（Room Obs），测试 AriGraph 世界模型是否能通过记住所有相关层级信息来弥补这一限制。 我们比较三个智能体。 第一个是文本观察受限的 NetPlay [Room obs]，第二个是接收 Room Obs 并更新 AriGraph 的 Ariadne [Room obs] 智能体，最后一个是能够访问已探索层级信息的 NetPlay [Level obs]。

4.3 Multi-hop Q&A

Although our memory architecture was originally designed for an agent interacting with the environment, we evaluated its performance on standard multi-hop Q&A benchmarks — Musique and HotpotQA to show its robustness and efficiency in more standard retrieval tasks. We made slight adjustments to the promts and replaced Contriever model with BGE-M3, as it is a better fit for general text encoding. We used 200 random samples from both datasets similar to Li et al. We compared the performance of our approach against Graphreader, ReadAgent, HOLMES, GraphRAG and RAG baselines provided in Li et al.

尽管我们的记忆架构最初是为与环境交互的智能体设计的，但为了展示它在更标准检索任务中的强健性和效率，我们在标准多跳问答基准 Musique 和 HotpotQA 上评估了它的表现。 我们对提示进行了轻微调整，并将 Contriever 模型替换为 BGE-M3，因为它更适合通用文本编码。 与 Li 等人类似，我们从两个数据集中各使用 200 个随机样本。 我们将我们方法的性能与 Graphreader、ReadAgent、HOLMES、GraphRAG 以及 Li 等人提供的 RAG 基线进行了比较。

5. Results

5.1 TextWorld

Every LLM based agent had five attempts to solve each game. The normalized score of one means that an agent completed the game, and score less than one represents intermediate progress. Results on text-based games are shown on the Figure 3 (for dynamics see Appendix). We estimate performance as average of three best runs. Ariadne successfully remembers and uses information about state of the world for all three tasks. Baseline agents are unable to solve the Treasure Hunt, and fail to find even second key in the Treasure Hunt Hardest. On the other hand, Ariadne successfully solves the Treasure Hunt in about fifty steps, maintains robust performance in the Treasure Hunt Hard, and is able to complete the Treasure Hunt Hardest with more then double amount of rooms compared to Hard version, additional keys and distractors (see Appendix).

每个基于 LLM 的智能体都有五次尝试来解决每个游戏。 归一化分数为 1 表示智能体完成了游戏，低于 1 的分数表示中间进展。 文本游戏上的结果见 图3（动态结果见附录）。 我们用三次最佳运行的平均值来估计性能。 Ariadne 在全部三个任务中都成功记住并使用了世界状态信息。 基线智能体无法解决 Treasure Hunt，在 Treasure Hunt Hardest 中甚至找不到第二把钥匙。 另一方面，Ariadne 能够在约五十步内成功解决 Treasure Hunt，在 Treasure Hunt Hard 中保持强健表现，并能够完成 Treasure Hunt Hardest；该版本相比 Hard 版本房间数超过两倍，还包含额外钥匙和干扰物（见附录）。

Compared to the Treasure Hunt, the Cleaning game possesses a slightly different challenge as it is more important to properly filter outdated information about object locations, than not to lose any information. This is evident from the reduced usefulness of Episodic Memory in Ariadne agent and Full history baseline, since both memory modules focus on retaining long-term information. Overall Ariadne notably outperforms alternatives in this game. Moreover, Ariadne also outperforms Reflexion, which has additional information between episodes. This baseline shows markable performance growth (in comparison wuth RAG) at the second try, but degrades with following tries.

与 Treasure Hunt 相比，Cleaning 游戏具有略微不同的挑战：正确过滤关于物品位置的过时信息，比不丢失任何信息更重要。 这一点可以从 Ariadne 智能体和 Full history 基线中情景记忆作用降低看出，因为这两个记忆模块都专注于保留长期信息。 总体而言，Ariadne 在该游戏中明显优于其他替代方案。 此外，Ariadne 还优于 Reflexion，后者在 episode 之间拥有额外信息。 该基线在第二次尝试时显示出明显性能增长（相对于 RAG），但在后续尝试中退化。

The Cooking game has the highest difficulty, because any error at intermediate step prevents completion of the whole game. All baseline agents (except Reflexion 2-shot with obvious advantage over other methods) fail to complete cooking tasks due to insufficient or misused information. In this game, episodic memory is particularly important, allowing the agent to recall useful observations such as the content of the recipe or cooking instructions. For token usage of every method see Appendix.

Cooking 游戏难度最高，因为中间步骤中的任何错误都会阻止整个游戏完成。 所有基线智能体（除了相对其他方法具有明显优势的 Reflexion 2-shot）都因为信息不足或信息使用不当而未能完成烹饪任务。 在该游戏中，情景记忆尤其重要，它使智能体能够回忆有用观察，例如食谱内容或烹饪说明。 各方法的 token 使用量见附录。

图4：Ariadne LLM 智能体在烹饪基准上相较 RL 替代方法展现出领先表现。

Comparison with RL baselines on variation of the Cooking task is shown in Figure 4. We run Ariadne and GPT-4 with Full history on 4 difficulty levels from the cooking benchmark. Ariadne shows superior performance to RL-agents on all 4 levels, especially harder ones. GPT-4 agent with Full history solves only first two levels which is consistent with previous result as the Cooking from Figure 3.A is harder than level 4.

与 RL 基线在 Cooking 任务变体上的比较见 图4。 我们在烹饪基准的 4 个难度等级上运行 Ariadne 和使用 Full history 的 GPT-4。 Ariadne 在全部 4 个等级上都优于 RL 智能体，尤其是在更困难的等级上。 使用 Full history 的 GPT-4 智能体只解决了前两个等级，这与先前结果一致，因为 图3.A 中的 Cooking 比 level 4 更难。

Human evaluation. Comparison with respect to human players is shown in Figure 3.C. All Humans is the average score of all valid (completed) human trials. Human Top-3 is the average score of three best plays for each task. Ariadne outperforms average human player from our sample on all tasks, and scores similarly to the best human plays in the Cooking and the Treasure Hunt, but underperforms in the Cleaning.

人类评估。 与人类玩家的比较见 图3.C。 All Humans 是所有有效（已完成）人类试验的平均分。 Human Top-3 是每个任务三次最佳游玩的平均分。 Ariadne 在所有任务上都优于我们样本中的平均人类玩家，并且在 Cooking 和 Treasure Hunt 中取得与最佳人类游玩相近的分数，但在 Cleaning 中表现较弱。

Graph quality. We measured AriGraph's growth rate and update rate during gameplay (see Figure 5). The graph actively grows during the exploration phase and flattens once the agent becomes familiar with the environment. We argue that this indicates that agent can generalize to long interactions with the environment despite constant updates to the semantic graph. Additional results in Appendix demonstrate that the growth rate of the graph decreases with the increase in quality of LLM backbone.

图质量。 我们测量了 AriGraph 在游戏过程中的增长率和更新率（见 图5）。 图在探索阶段积极增长，一旦智能体熟悉环境后就趋于平缓。 我们认为这表明，尽管语义图持续更新，智能体仍可以泛化到与环境的长时间交互。 附录中的额外结果表明，随着 LLM backbone 质量提高，图的增长率会下降。

图5：AriGraph 在学习过程中以及随环境规模增长时展现出良好可扩展性。知识图谱大小在探索和学习阶段很快饱和；当 Treasure Hunt 和 Cooking 的困难版本包含更多房间与物体时，KG 适度增长。

Overal results demonstrate clear advantage of Ariadne agent over LLM based and RL baselines. Semantic memory enables the Ariadne Agent to build and update knowledge about the current state of the POMDP environment, which is crucial for navigation, exploration and capturing relevant details in interactive environments. On the other hand, episodic memory assists the agent in retrieving detailed long-term information that may not be captured in semantic memory, as demonstrated by the results in the Cooking task.

总体结果显示，Ariadne 智能体相对于基于 LLM 和 RL 的基线具有明确优势。 语义记忆使 Ariadne 智能体能够构建并更新关于 POMDP 环境当前状态的知识，这对于导航、探索以及捕捉交互环境中的相关细节至关重要。 另一方面，情景记忆帮助智能体检索语义记忆中可能未捕获的详细长期信息，这一点由 Cooking 任务的结果所展示。

5.2 NetHack

The results are presented in Table 1. Scores column shows average game score across 3 runs, Levels column shows average number of dungeon levels completed by an agent. GPT-4o was used for all agents. Underscoring the importance of memory in this task, NetPlay [Level obs] with access to memory oracle achieved the highest scores, while NetPlay [Room obs] with only current room observations performed the worst. Ariadne [Room obs] successfully utilized AriGraph word model, achieving performance comparable to the baseline with memory oracle.

结果见 表1。 Scores 列显示 3 次运行的平均游戏分数，Levels 列显示智能体完成的平均地牢层数。 所有智能体都使用 GPT-4o。 NetPlay [Level obs] 可以访问记忆 oracle，并取得最高分；而仅有当前房间观察的 NetPlay [Room obs] 表现最差，这凸显了记忆在该任务中的重要性。 Ariadne [Room obs] 成功利用 AriGraph 世界模型，实现了与带有记忆 oracle 的基线相当的表现。

表1：Ariadne 在遮蔽的部分观察下，表现可与拥有完整层级信息的 NetPlay 智能体相当。

Method	Score	Levels
Ariadne (Room obs)	$593.00 \pm 202.62$	$6.33 \pm 2.31$
NetPlay (Room obs)	$341.67 \pm 109.14$	$3.67 \pm 1.15$
NetPlay (Level obs)	$675.33 \pm 130.27$	$7.33 \pm 1.15$

5.3 Multi-hop Q&A

We compared AriGraph with the latest LLM-based approaches that employ knowledge graph construction and retrieval techniques for question answering over documents (Table 2). Our memory architecture, adapted from the Ariadne TextWorld agent, utilizing both GPT-4 and GPT-4o-mini outperformed baseline methods like ReadAgent (GPT-4), GPT-4 RAG, GPT-4 full context and GraphReader (GPT-4). GraphRAG served as a strong GPT-4o-mini baseline, due to its extremely hight costs. ArigGraph (GPT-4o-mini) showed weaker performance on Musique, but outperformed GraphRAG on HotpotQA. Notably, our approach is more then 10x cheaper in comparison to GraphRAG (Table in Appendix).

我们将 AriGraph 与最新的基于 LLM 的方法进行了比较，这些方法使用知识图谱构建与检索技术来进行文档问答（表2）。 我们的记忆架构改编自 Ariadne TextWorld 智能体，并同时使用 GPT-4 和 GPT-4o-mini；它优于 ReadAgent (GPT-4)、GPT-4 RAG、GPT-4 full context 和 GraphReader (GPT-4) 等基线方法。 由于 GraphRAG 成本极高，它作为一个强 GPT-4o-mini 基线。 ArigGraph (GPT-4o-mini) 在 Musique 上表现较弱，但在 HotpotQA 上优于 GraphRAG。 值得注意的是，与 GraphRAG 相比，我们的方法成本低 10 倍以上（见附录表）。

The best performance using GPT-4 was achieved by HOLMES, but AriGraph (GPT-4) exhibited comparable results. Notably, all baseline methods were specifically designed for Q&A tasks, incorporating task-specific prompt tuning and additional architectural enhancements. Both GraphRAG and HOLMES employ hyper-relations in their graphs to connect source data with extracted entities, similar to our method. However, these approaches lack mechanisms for updates in dynamic environments, a key advantage of AriGraph.

使用 GPT-4 的最佳性能由 HOLMES 达成，但 AriGraph (GPT-4) 展现出相当的结果。 值得注意的是，所有基线方法都是专门为 Q&A 任务设计的，包含任务特定提示调优和额外架构增强。 GraphRAG 和 HOLMES 都在其图中使用超关系，将源数据与抽取实体连接起来，这与我们的方法类似。 然而，这些方法缺少在动态环境中更新的机制，而这是 AriGraph 的关键优势。

表2：AriGraph 记忆在 Multi-Hop Q&A 数据集上表现出竞争力。即使在非交互任务中，AriGraph 也可与强 QA 基线智能体相当。GPT-4 与 GPT-4o-mini 下的最佳结果分别用粗体和下划线表示。

Method	MuSiQue		HotpotQA
	EM	F1	EM	F1
BM25(top-3)	25.0	31.1	45.7	58.5
Ada-002(top-3)	24.5	32.1	45.0	58.1
GPT-4 full context	33.5	42.7	53.0	68.4
GPT-4 + supporting facts	45.0	56.0	57.0	73.8
ReadAgent(GPT-4)	35.0	45.1	48.0	62.0
GraphReader(GPT-4)	38.0	47.4	55.0	70.0
HOLMES(GPT-4)	48.0	58.0	66.0	78.0
AriGraph(GPT-4)	45.0	57.0	68.0	74.7
GraphRAG(GPT-4o-mini)	40.0	53.5	58.7	63.3
AriGraph(GPT-4o-mini)	36.5	47.9	60.0	68.6

6. Related work

Voyager, Ghost in the Minecraft and Jarvis-1 are advanced, open-ended LLM agents that show significantly better performance in Minecraft compared to earlier techniques. These agents feature memory capabilities through a library of learned skills, summaries of successful actions, and episodic memory with plans for successful task execution. However, they fall short in representing knowledge with semantic structure and depend heavily on the LLM's extensive Minecraft knowledge, or even access to the Minecraft wiki. Generative agents mimic human behavior in multi-agent environments and were among the pioneers in introducing an advanced memory system for LLM agents, which we use as our baseline. Reflexion and CLIN enables agents to reflect on past trajectories, allowing them to store relevant insights about completed actions in a long-term memory module, but has no structural representation of knowledge and episodic memories. LARP utilizes the concepts of episodic and semantic memories but treats them as separate instances and lacks a structural representation of knowledge.

Voyager、Ghost in the Minecraft 和 Jarvis-1 是先进的开放式 LLM 智能体，相比早期技术，它们在 Minecraft 中表现显著更好。 这些智能体通过已学习技能库、成功动作总结，以及带有成功任务执行计划的情景记忆来具备记忆能力。 然而，它们在用语义结构表示知识方面不足，并且严重依赖 LLM 广泛的 Minecraft 知识，甚至依赖访问 Minecraft wiki。 Generative agents 在多智能体环境中模拟人类行为，是最早为 LLM 智能体引入高级记忆系统的工作之一，我们将其用作基线。 Reflexion 和 CLIN 使智能体能够反思过去轨迹，从而在长期记忆模块中存储关于已完成动作的相关洞察，但它们没有知识和情景记忆的结构化表示。 LARP 利用情景记忆和语义记忆概念，但将它们视为独立实例，并缺少知识的结构化表示。

Considerable research is dedicated to leveraging established knowledge graphs for enhancing Q&A systems to address the factual knowledge deficiency observed in LLMs. The latest research demonstrating best performance in Q&A tasks includes Graphreader, HOLMES, HippoRAG, GraphRAG which all employ the technique of building knowledge graphs from texts. However, these studies do not address the context of functioning within an interactive environment, nor do they take into account the updates to knowledge graphs prompted by new experiences.

大量研究致力于利用已有知识图谱增强 Q&A 系统，以解决 LLM 中观察到的事实知识不足问题。 在 Q&A 任务上展现最佳表现的最新研究包括 Graphreader、HOLMES、HippoRAG 和 GraphRAG，它们都采用从文本构建知识图谱的技术。 然而，这些研究并未处理在交互环境中运行的情境，也未考虑由新经验触发的知识图谱更新。

Text-based environments were originally designed to evaluate reinforcement learning (RL) agents. Multiple experiments have already explored the potential of LLMs in these complex scenarios. However raw LLMs show poor results in these games without proper agentic architecture and memory.

文本环境最初是为评估强化学习（RL）智能体而设计的。 多个实验已经探索了 LLM 在这些复杂场景中的潜力。 然而，如果没有合适的智能体架构和记忆，原始 LLM 在这些游戏中表现很差。

7. Conclusions

In this paper, we introduced AriGraph, a novel knowledge graph world model tailored for LLM agents. AriGraph uniquely integrates semantic and episodic memories from textual observations, providing a structured and dynamic representation of knowledge. We evaluated this approach across a range of interactive text-based games and multi-hop Q&A benchmarks, comparing it against existing memory architectures. To test its capabilities comprehensively, we developed a cognitive architecture called Ariadne, which combines AriGraph with planning and decision-making components.

在本文中，我们介绍了 AriGraph，这是一种为 LLM 智能体量身设计的新型知识图谱世界模型。 AriGraph 以独特方式整合来自文本观察的语义记忆和情景记忆，提供结构化且动态的知识表示。 我们在一系列交互式文本游戏和多跳 Q&A 基准上评估了该方法，并将其与现有记忆架构进行比较。 为了全面测试其能力，我们开发了一种名为 Ariadne 的认知架构，将 AriGraph 与规划和决策组件结合起来。

Our results demonstrate that AriGraph significantly outperforms other memory systems in tasks requiring long-term memory, such as decision-making, planning, and exploration in partially observable environments. The structured knowledge representation provided by AriGraph enables efficient retrieval and reasoning, accelerating learning and task completion. Additionally, AriGraph’s scalability was evident as it maintained high performance even when the complexity of tasks increased, involving more objects and locations. In multi-hop Q&A benchmarks, AriGraph exhibited competitive performance, underscoring its robustness and adaptability beyond interactive environments.

我们的结果表明，在需要长期记忆的任务中，例如部分可观察环境中的决策、规划和探索，AriGraph 显著优于其他记忆系统。 AriGraph 提供的结构化知识表示支持高效检索和推理，从而加速学习和任务完成。 此外，AriGraph 的可扩展性也很明显：即使任务复杂度提高、涉及更多物体和地点，它仍能保持高性能。 在多跳 Q&A 基准上，AriGraph 展现出竞争性表现，凸显了它在交互环境之外的强健性和适应性。

While promising, our approach can be further enhanced by incorporating multi-modal observations, procedural memories, and more sophisticated graph search methods.

尽管前景可观，我们的方法仍可以通过引入多模态观察、程序性记忆和更复杂的图搜索方法进一步增强。