---

A Notion of Complexity for Theory of Mind via Discrete World Models

Huang X A, La Malfa E, Marro S, et al. A Notion of Complexity for Theory of Mind via Discrete World Models. Findings of the Association for Computational Linguistics: EMNLP 2024.

arXiv · GitHub

EMNLP 2024 FindingsTheory of MindLLM EvaluationPromptingDWMBologna / Oxford / Alan Turing / Leeds

---

心智理论复杂度的一种定义：通过离散世界模型

Abstract

Theory of Mind (ToM) can be used to assess the capabilities of Large Language Models (LLMs) in complex scenarios where social reasoning is required. While the research community has proposed many ToM benchmarks, their hardness varies greatly, and their complexity is not well defined. This work proposes a framework inspired by cognitive load theory to measure the complexity of ToM tasks. We quantify a problem's complexity as the number of states necessary to solve it correctly.

心智理论（ToM）可用于评估大语言模型（LLM）在需要社会推理的复杂场景中的能力。 尽管研究社区已经提出许多 ToM 基准，但它们的难度差异很大，其复杂度也没有得到清晰定义。 本工作提出一个受认知负荷理论启发的框架，用于衡量 ToM 任务的复杂度。 我们将问题的复杂度量化为正确解决该问题所需的状态数量。

Our complexity measure also accounts for spurious states of a ToM problem designed to make it apparently harder. We use our method to assess the complexity of five widely adopted ToM benchmarks. On top of this framework, we design a prompting technique that augments the information available to a model with a description of how the environment changes with the agents' interactions. We name this technique Discrete World Models (DWM) and show how it elicits superior performance on ToM tasks.

我们的复杂度度量还考虑了 ToM 问题中为了使其看起来更难而设计的伪状态。 我们使用该方法评估五个广泛采用的 ToM 基准的复杂度。 在这一框架之上，我们设计了一种提示技术，用关于环境如何随智能体交互而变化的描述来扩充模型可获得的信息。 我们将这项技术命名为离散世界模型（Discrete World Models，DWM），并展示它如何在 ToM 任务上引出更优的性能。

1. Introduction

图1：经典 Sally-Anne 问答任务上的 DWM 提示示例。DWM 将原任务划分为一系列状态事件，并提示 LLM 描述状态；在多数情况下，这有助于 LLM 给出正确答案。

Theory of Mind (ToM) studies how agents form and use beliefs to reason in dynamic environments. Originally developed to describe human interactions as well as toddlers' psychological development, ToM has been quickly adopted by other fields, including artificial intelligence, bayesian inference and machine learning. In machine learning, ToM has both descriptive and prescriptive usage: on the one hand, ToM benchmarks assess the capabilities of a model in complex environments; on the other, ToM's frameworks such as theory-theory and simulation theory have been widely adopted to test the proficiency of Large Language Models (LLMs) in social tasks where humans excel.

心智理论（ToM）研究智能体如何形成并运用信念，在动态环境中进行推理。 ToM 最初用于描述人类互动以及幼儿的心理发展，随后很快被人工智能、贝叶斯推断和机器学习等其他领域采用。 在机器学习中，ToM 兼具描述性与规范性用途：一方面，ToM 基准评估模型在复杂环境中的能力；另一方面，理论论（theory-theory）和模拟理论（simulation theory）等 ToM 框架被广泛用于测试大语言模型（LLM）在人类擅长的社会任务中的熟练程度。

In this work, we propose a framework to characterise a ToM benchmark's difficulty, i.e., its complexity, as the number of state events that are sufficient to track the state of an object, including $k^{\text{th}}$-order beliefs motivated by theoretical parallelisms with Sweller's cognitive load theory.

在本工作中，我们提出一个刻画 ToM 基准难度的框架，即将其复杂度定义为足以追踪对象状态的状态事件数量，其中包括受 Sweller 认知负荷理论之理论对应关系启发的 $k^{\text{th}}$ 阶信念。

We characterise the complexity of five standard ToM benchmarks, from false belief to commonsense and social reasoning, and compute their complexity as a proxy of their inherent difficulty. Inspired by prompting techniques that split a task into elementary sub-problems that are solved sequentially, like Tree of Thoughts and least-to-most prompting, we introduce a technique that stimulates a model's reasoning capabilities via Discrete World Models (DWM). DWM leverages the notion of statefulness via a succinct and coherent representation of each state events, as illustrated in Figure 1. We test DWM on ToMi, MindGames, Adv-CSFB, SocialIQA, and FANToM, eliciting superior performance than Chain of Thoughts (CoT) and Tree of Thoughts (ToT) on those problems whose state spaces are informative.

我们刻画五个标准 ToM 基准的复杂度，任务范围从错误信念延伸到常识与社会推理，并以所计算的复杂度作为其内在难度的代理指标。 受思维树和由简到繁提示等将任务拆分为依次求解的基本子问题的提示技术启发，我们提出一种通过离散世界模型（DWM）激发模型推理能力的技术。 如图1所示，DWM 通过对每个状态事件给出简洁且连贯的表示来利用状态性概念。 我们在 ToMi、MindGames、Adv-CSFB、SocialIQA 和 FANToM 上测试 DWM；对于状态空间具有信息量的问题，它引出的性能优于思维链（CoT）和思维树（ToT）。

We further assess whether memorisation affects a model's performance, and we discover that while this phenomenon happens for standard benchmarks such as ToMi, with input-output pairs that can be retrieved word for word via prompting, it does not strongly correlate with a drop of performance on memorised ToM benchmarks. We conduct our experiments on a variety of open- and closed-source LLMs, including GPT-3.5-Turbo, GPT-4, LLaMA3-70B and Mixtral 8x7B. In summary, in this paper:

我们进一步评估记忆是否会影响模型性能，并发现这种现象虽然会出现在 ToMi 等标准基准中，其输入输出对可通过提示被逐字检索出来，但它与模型在已记忆 ToM 基准上的性能下降并没有很强的相关性。 我们在多种开源和闭源 LLM 上开展实验，包括 GPT-3.5-Turbo、GPT-4、LLaMA3-70B 和 Mixtral 8x7B。 总而言之，本文的贡献如下：

• We introduce the concept of complexity of a ToM task to quantify the hardness of keeping track of the elements (e.g., agents' beliefs or objects' states) that are sufficient to produce the correct answer to different problems inspired by frameworks in cognitive science.

• We propose DWM, a simple yet effective prompting technique that improves a model's capability by making implicit information explicit while not necessitating exogenous information (i.e., it does not require RAG or fine-tuning).

• 我们引入 ToM 任务的复杂度概念，用于量化追踪那些足以为不同问题生成正确答案的要素（例如智能体信念或对象状态）的难度；该概念受认知科学框架启发。

• 我们提出 DWM，这是一种简单而有效的提示技术，通过使隐式信息显式化来提升模型能力，同时不需要外源信息（即不需要 RAG 或微调）。

We consider our work a step towards a framework that formalizes the hardness of a ToM problem in an unambiguous way, inspired by the theory of World Models.

我们认为，本工作朝着以无歧义方式形式化 ToM 问题难度的框架迈出了一步，其灵感来自世界模型理论。

2. Related Work

Over 40 years of research on ToM in psychology on human development has created a fertile ground for the development of these ideas in adjacent fields. In the last decade, many works studied ToM in artificial intelligence and machine learning, with applications to multi-agent systems and reinforcement learning. More recently, the rise in popularity of LLMs shifted the interest towards understanding and benchmarking large models' capacity to solve increasingly complex ToM tasks. While some researchers believe LLMs have already become proficient in solving ToM tasks, others show scepticism and illustrate cases where they fail on trivial variations of well-known problems.

心理学领域四十多年来围绕人类发展的 ToM 研究，为这些思想在相邻领域的发展创造了沃土。 过去十年中，许多工作研究了人工智能和机器学习中的 ToM，并将其应用于多智能体系统和强化学习。 近年来，LLM 日益流行，研究兴趣随之转向理解和评测大型模型解决日益复杂的 ToM 任务的能力。 一些研究者认为 LLM 已经能够熟练解决 ToM 任务，另一些研究者则持怀疑态度，并展示了模型在知名问题的微小变体上失败的案例。

In a joint effort between computer scientists and psychologists, many ToM benchmarks have been developed and used to test neural-network models, including LLMs. Recently, concepts such as World Models have found applicability mostly as discrete prompting techniques in conjunction with optimisation procedures. Researchers have found evidence of an emergent internal representation (e.g., World Model's surrogates) of the state games and state-tracking abilities, necessary for correct belief tracking in ToM problems.

在计算机科学家和心理学家的共同努力下，许多 ToM 基准被开发出来，并用于测试包括 LLM 在内的神经网络模型。 近年来，世界模型等概念主要以离散提示技术结合优化过程的形式得到应用。 研究者发现了游戏状态的涌现式内部表示（例如世界模型的替代表示）以及状态追踪能力的证据，而这些能力是 ToM 问题中正确追踪信念所必需的。

Cognitive load theory emerged in the late eighties with Sweller's work on human problem solving. Most measures of cognitive load are based on subjective reports from humans. Even though some attempts at automatic cognitive load measures have been present, they have not been widely adopted in the community. The works that are more similar to our complexity framework are only tangentially related to ToM. Inspired by the work in Zhou et al. and the results in Zhou et al., our prompting technique is inspired by Park et al. and Nye et al.: the former develops an architecture to record the agent's experiences. The latter proposes a prompting technique that forces a model to express the intermediate computational steps to solve a problem.

认知负荷理论兴起于 20 世纪 80 年代末 Sweller 关于人类问题求解的研究。 大多数认知负荷度量都基于人类的主观报告。 尽管已经出现一些自动度量认知负荷的尝试，但它们尚未被社区广泛采用。 与我们的复杂度框架更相似的工作只与 ToM 间接相关。 受 Zhou 等人的工作和实验结果启发，我们的提示技术还借鉴了 Park 等人和 Nye 等人的研究：前者开发了一种记录智能体经历的架构。 后者提出一种提示技术，迫使模型表达解决问题时的中间计算步骤。

图2：对图1示例计算状态性与无状态性的过程。对 obj1 而言，用于追踪苹果的最优划分合并了前两个状态和输入提示片段；对涉及 Bob 一阶信念的 obj2 而言，状态性更高。底部按照复杂度定义计算任务复杂度，与问答不直接相关的对象复杂度会被折扣。

3. Methodology

In this section, we introduce a notion of complexity for ToM problems which quantifies the hardness of a problem as the number of computational steps humans take to solve them and compare it with Sweller's cognitive load theory. We then present the DWM prompting technique within the complexity framework and show how it differs from standard methods like CoT and ToT. We further characterise its efficiency with the number of input/output tokens and queries to a model as the control variables.

在本节中，我们引入一种 ToM 问题复杂度定义，将问题难度量化为人类求解时采取的计算步骤数量，并将其与 Sweller 的认知负荷理论进行比较。 随后，我们在复杂度框架内介绍 DWM 提示技术，并说明它与 CoT 和 ToT 等标准方法有何不同。 我们还以输入输出 token 数量和模型查询次数为控制变量，进一步刻画其效率。

3.1 On the Complexity of ToM

The need to provide a consistent representation of the environment, including each agent's beliefs, inspired us to characterise the complexity of a ToM problem in terms of sufficient elements to track to output the correct result. Consider a problem prompt $p$, expressed in natural language, that describes how multiple agents interact with an environment object $\mathbf{obj}$, as illustrated in Figure 2 (top). In our framework, an object can be the state of the apple as well as the $k^{\text{th}}$-order belief of an agent about the apple position. Our framework naturally extends to multiple objects by considering their union.

为了给出包含每个智能体信念在内的一致环境表示，我们受到启发，以输出正确结果时需要追踪的充分要素来刻画 ToM 问题的复杂度。 考虑一个以自然语言表达的问题提示 $p$，它描述多个智能体如何与环境对象 $\mathbf{obj}$ 交互，如图2顶部所示。 在我们的框架中，对象既可以是苹果的状态，也可以是智能体关于苹果位置的 $k^{\text{th}}$ 阶信念。 通过考虑多个对象的并集，我们的框架可以自然扩展到多对象情形。

Suppose that in $p$, the state of $\mathbf{obj}$ is modified $T>0$ times, thus identifying $T$ unique configurations, namely $E_{\mathbf{obj}}=\{e_1,..,e_T\}$. To correctly solve a ToM task where $p$ is complemented by a query about $\mathbf{obj}$, a model should distinguish between the interactions that modify the configuration of $\mathbf{obj}$, i.e., the stateful states, from those that modify any other stateless object $Obj\setminus \mathbf{obj}$, i.e., those that one does not need to track.

假设在 $p$ 中，$\mathbf{obj}$ 的状态被修改了 $T>0$ 次，从而确定 $T$ 个唯一配置，即 $E_{\mathbf{obj}}=\{e_1,..,e_T\}$。 为了正确求解由 $p$ 和一个关于 $\mathbf{obj}$ 的查询构成的 ToM 任务，模型应当区分修改 $\mathbf{obj}$ 配置的交互，即有状态状态，与修改其他任意无状态对象 $Obj\setminus \mathbf{obj}$ 的交互，即无需追踪的交互。

We first show how to define the cost of tracking a task's stateful states, which we complement with that of the stateless. Both definitions concur in defining the complexity of a ToM task.

我们首先说明如何定义追踪任务有状态状态的成本，再以无状态部分的成本对其进行补充。 这两个定义共同构成 ToM 任务的复杂度定义。

3.1.1 Stateful and Stateless Complexity

For a ToM task, expressed as $p$, that describes the evolution of an environment where an unknown number of atomic iterations $T$ modifies $\mathbf{obj}$ or its perception, each environment state $e_t\in E_{\mathbf{obj}}$ can be coupled with the prompt prefix $p_{\le t}\text{ s.t. }{p}_{\le t}\oplus p_{>t}=p$, that describes such configuration. We denote $(e_t,p_{\le t})$ as a generic state description, as illustrated in Figure 2 (top).

对于以 $p$ 表示的 ToM 任务，它描述一个由未知数量的原子交互 $T$ 修改 $\mathbf{obj}$ 或其感知的环境演化过程；每个环境状态 $e_t\in E_{\mathbf{obj}}$ 都可与描述该配置的提示前缀 $p_{\le t}\text{ s.t. }{p}_{\le t}\oplus p_{>t}=p$ 配对。 我们将 $(e_t,p_{\le t})$ 记为通用的状态描述，如图2顶部所示。

Definition 1 (State event). A state event for an object $\mathbf{obj}$ is an event that links adjacent state descriptions that involve, for both the environment state $e_t$ and the sub-prompt $p_{\le t}$, a state change of $\mathbf{obj}$. Formally, we define a relation, $F_{\mathbf{obj}}$, to specify which pairs of state descriptions form a state event:

**定义 1（状态事件）。**对象 $\mathbf{obj}$ 的状态事件是连接相邻状态描述的事件；对于环境状态 $e_t$ 和子提示 $p_{\le t}$，这些描述都涉及 $\mathbf{obj}$ 的状态变化。 形式化地，我们定义关系 $F_{\mathbf{obj}}$，用于指定哪些状态描述对构成状态事件：

$$\begin{array}{rl}{F}_{\mathbf{obj}}((e_t,p_{\le t}),(e_{t+1},p_{\le t+1}))\equiv & e_t\ne e_{t+1}\\ & \wedge p_{\le t+1}=p_{\le t}\oplus p_{t+1}.\end{array}$$

Here $1\le t\le |p|$ ($|p|$ denotes the number of atomic prompts) and $\oplus$ is the string concatenation operator. Thus a state event $F_{\mathbf{obj}}$ identifies those state descriptions $(e_t,p_{\le t})$ which have a successor $(e_{t+1},p_{\le t+1})$ where $\mathbf{obj}$ has changed its configuration.

其中 $1\le t\le |p|$（$|p|$ 表示原子提示的数量），$\oplus$ 是字符串拼接算子。 因此，状态事件 $F_{\mathbf{obj}}$ 标识那些具有后继 $(e_{t+1},p_{\le t+1})$ 的状态描述 $(e_t,p_{\le t})$，并且 $\mathbf{obj}$ 在该后继中改变了配置。

In the context of ToM tasks, a state event could be a person who moves an object, exits (thus introducing partial observability) or witnesses a change in the environment (as now the description of the environment will take that change into account), as illustrated in Figure 2 (middle). Our prompting technique, namely DWM (Section 3.2), aims at making implicit observations about objects explicit.

在 ToM 任务中，状态事件可以是某人移动对象、离开（从而引入部分可观测性），或目睹环境发生变化（因为此后的环境描述会考虑这一变化），如图2中部所示。 我们的提示技术 DWM（第 3.2 节）旨在使关于对象的隐式观察显式化。

We finally introduce the notion of partition function to connect the maximum number of non-empty state events relative to a prompt. Such a notion will serve as the building block to compute the complexity of a ToM problem.

最后，我们引入划分函数概念，用以联系相对于一个提示的非空状态事件的最大数量。 这一概念将作为计算 ToM 问题复杂度的基础模块。

Definition 2 (Partitions). A partition $par{t}_{\mathbf{obj}}$ w.r.t. $\mathbf{obj}$ identifies those state events which partition a ToM prompt $p$ into sequential segments where $\mathbf{obj}$ changes its value. Formally:

**定义 2（划分）。**相对于 $\mathbf{obj}$ 的划分 $par{t}_{\mathbf{obj}}$ 标识那些将 ToM 提示 $p$ 划分为顺序片段的状态事件，$\mathbf{obj}$ 会在这些片段间改变取值。 形式化地：

$$\begin{array}{rl}\text{Let }par{t}_{\mathbf{obj}}=\{(e_t,p_{\le t}):& F_{\mathbf{obj}}((e_t,p_{\le t}),\\ & (e_{t+1},p_{\le t+1}))\\ & \wedge e_t\in E_{\mathbf{obj}}\}.\end{array}$$

Definition 2 describes an optimal partition, $par{t}_{\mathbf{obj}}$ of state descriptions that covers all the relevant changes to $\mathbf{obj}$. The partition is represented by the set of event descriptions where $\mathbf{obj}$ changes its description immediately after. Note that this set of event descriptions is unique for any $\mathbf{obj}$.

定义 2 描述了状态描述的最优划分 $par{t}_{\mathbf{obj}}$，它覆盖 $\mathbf{obj}$ 的所有相关变化。 该划分由一组事件描述表示，$\mathbf{obj}$ 会在每个事件描述之后立即改变其描述。 请注意，对于任意 $\mathbf{obj}$，这组事件描述都是唯一的。

3.1.2 The Complexity of a ToM Task

We can now define the notion of statefulness of a ToM task specified as a prompt $p$ as the size of Definition 2, namely $T_{\mathbf{obj}}=|par{t}_{\mathbf{obj}}|$. The process of computing the statefulness of an object or its belief is illustrated in Figure 2.

现在，我们可以将以提示 $p$ 指定的 ToM 任务的状态性定义为定义 2 中集合的大小，即 $T_{\mathbf{obj}}=|par{t}_{\mathbf{obj}}|$。 对象或其信念的状态性计算过程如图2所示。

For a ToM task where the question to solve relates to an object $\mathbf{obj}$, one must ensure that changes to any other object, namely $Obj\setminus \mathbf{obj}$, do not affect $\mathbf{obj}$. While tracking the evolution of what is irrelevant to answer the question is unnecessary, a computation model must assess whether a particular environmental change affected $\mathbf{obj}$. We thus introduce the notion of statelessness, i.e., the cost of discerning whether a change in the environment affects $\mathbf{obj}$. The computation is similar to that of Definition 2, except that $\mathbf{obj}$ is replaced with any object in $Obj\setminus \mathbf{obj}$; however for stateless objects, we introduce a discount factor $\tau$ to penalise the complexity of state events that do not affect $\mathbf{obj}$. Mathematically, we formalise the statelessness of a ToM task involving an object $\mathbf{obj}$ as $\tau \sum _{obj\in Obj\setminus \mathbf{obj}}{T}_{obj}$.

对于问题所询问对象为 $\mathbf{obj}$ 的 ToM 任务，必须确保其他任意对象，即 $Obj\setminus \mathbf{obj}$，发生的变化不会影响 $\mathbf{obj}$。 虽然没有必要追踪与回答问题无关的对象演化，但计算模型必须判断特定环境变化是否影响了 $\mathbf{obj}$。 因此，我们引入无状态性概念，即辨别环境变化是否影响 $\mathbf{obj}$ 的成本。 其计算与定义 2 类似，只是将 $\mathbf{obj}$ 替换为 $Obj\setminus \mathbf{obj}$ 中的任意对象；然而，对于无状态对象，我们引入折扣因子 $\tau$，以降低不影响 $\mathbf{obj}$ 的状态事件所贡献的复杂度。 在数学上，我们将涉及对象 $\mathbf{obj}$ 的 ToM 任务的无状态性形式化为 $\tau \sum _{obj\in Obj\setminus \mathbf{obj}}{T}_{obj}$。

Finally, we formalise the complexity of a ToM task w.r.t. an object $\mathbf{obj}$ as the complexity of the stateful states plus the (discounted) sum of the others (i.e., stateless). Namely:

最后，我们将 ToM 任务相对于对象 $\mathbf{obj}$ 的复杂度形式化为有状态状态的复杂度，加上其他状态（即无状态部分）的折扣复杂度之和。 即：

$$T_{\mathbf{obj}}+\tau \sum _{obj\in Obj\setminus \mathbf{obj}}{T}_{obj}.$$

The process of computing the complexity of a ToM task is illustrated in Figure 2.

ToM 任务复杂度的计算过程如图2所示。

图3：左：图1示例中的 DWM 提示。我们以交互方式向 LLM 提供 ToM 问题，并要求它给出每个智能体信念的简洁表示。右：DWM 方法示意。输入首先被分成 T 个状态描述；随后，LLM 为每一部分描述环境状态事件及其变化；最后，所有输入片段和描述被送入 LLM 以获得任务答案。

3.1.3 A parallelism from the Cognitive sciences

Understanding how humans solve complex problems has long served as a valuable source of inspiration for advancing machine intelligence. Thought frameworks in the cognitive sciences, such as Kahneman's Dual-process theory, have greatly influenced various fields, including artificial intelligence. In this work, we draw ideas from another theory, less known in the community, the cognitive load theory. According to this theory, three main factors influence the mental effort humans exert when solving a particular task or learning new information: intrinsic, extraneous and germane load.

理解人类如何解决复杂问题，长期以来一直是推动机器智能发展的重要灵感来源。 认知科学中的思维框架，例如 Kahneman 的双过程理论，已经深刻影响了包括人工智能在内的多个领域。 在本工作中，我们还借鉴了社区中较少为人所知的另一种理论，即认知负荷理论。 根据该理论，人类在解决特定任务或学习新信息时付出的心理努力主要受三个因素影响：内在负荷、外在负荷和相关负荷。

The intrinsic load measures the natural difficulty of a certain task, the information that needs to be digested before answering the question. This relates to the complexity of the material itself. The extraneous load concerns the presentation of the information in the problem. For example, if questions phrased in a complex manner are used with a child, it would be much more difficult to understand and answer correctly compared to easier phrasing. Similarly, if many confounding sentences or sentences that do not matter in answering a question are present in the text, we expect an LLM to be worse, suggesting a weak similarity between the means of reason of these models and humans.

内在负荷衡量特定任务的自然难度，即在回答问题前必须消化的信息。 它与材料本身的复杂度有关。 外在负荷涉及问题中信息的呈现方式。 例如，如果向儿童提出措辞复杂的问题，那么与更简单的措辞相比，理解并正确回答会困难得多。 类似地，如果文本中存在许多混淆句或与回答问题无关的句子，我们预期 LLM 的表现会更差，这表明这些模型与人类的推理方式之间存在某种较弱的相似性。

Finally, germane load estimates the working memory resources needed to understand the important parts of the problem, i.e. the intrinsic load. If part of the memory is devoted to the extraneous load, then the germane load is diminished, suggesting a positive correlation with intrinsic load and a negative correlation with extraneous load.

最后，相关负荷估计理解问题重要部分，即内在负荷，所需的工作记忆资源。 如果部分记忆被用于处理外在负荷，那么相关负荷就会减少，这意味着它与内在负荷正相关、与外在负荷负相关。

3.1.4 A comparison with the Cognitive Load Theory

Our framework, summarized in Figure 2, has two main parts: stateful and stateless complexity. These notions have some similarities with, respectively, the intrinsic load and extraneous load. Stateful complexity provides a measure on the sentences that are needed to answer the question correctly and must be adequately represented in memory. In a similar manner, intrinsic load concerns on the needed information to correctly analyze a task. Likewise stateless complexity yields information about the confounding or irrelevant sentences and phrases in the text akin to extraneous load.

我们的框架总结于图2，它包含两个主要部分：有状态复杂度和无状态复杂度。 这两个概念分别与内在负荷和外在负荷存在一些相似之处。 有状态复杂度度量正确回答问题所需、且必须在记忆中得到充分表示的句子。 类似地，内在负荷关注正确分析任务所需的信息。 同样，无状态复杂度提供关于文本中混淆或无关句子与短语的信息，这与外在负荷相似。

In our setting, germane load could be interpreted as the ratio of the stateful and stateless complexity: higher ratio means higher density of useful sentences in answering a question. This notion of load could be used as a basis of an objective measure on the quality of a question-answering sample: given the same quantity of cognitive load, i.e. complexity, we would like to have a simple presentation with correct information, maximizing the germane load. If the cognitive load hypothesis applies to LLMs, maximizing the germane load would lower the complexity of the tasks given to a model, and therefore it would aid the model to answer questions more accurately.

在我们的设定中，相关负荷可被解释为有状态复杂度与无状态复杂度之比：比值越高，回答问题时有用句子的密度越高。 这一负荷概念可以作为客观度量问答样本质量的基础：在认知负荷，即复杂度，相同的情况下，我们希望信息呈现简单且正确，从而最大化相关负荷。 如果认知负荷假设适用于 LLM，那么最大化相关负荷会降低交给模型的任务复杂度，从而帮助模型更准确地回答问题。

3.2 Discrete World Models

We first introduce the background notation for prompting LLMs and assessing their accuracy on a standard classification task. We then propose our technique, namely DWM, which we eventually connect with the notion of statefulness of a ToM task.

我们首先介绍用于提示 LLM 并评估其在标准分类任务上准确率的背景符号。 随后，我们提出 DWM 技术，并最终将它与 ToM 任务的状态性概念联系起来。

Background notation. A (Large) Language Model is a function that predicts the next token (out of a finite vocabulary) conditioned on the sequence of previously fed/generated tokens, namely $\psi :\mathbf{v}\in V^{\ast }\to v\in V$. Such a mechanism can be used to sample multiple token outputs until an end-of-text token is predicted, by invoking $\psi$ in an auto-regressive fashion, i.e., $\psi (v\mid \mathbf{v})$. In our setting, a problem is specified as a tuple $(p,Q)$, where $p$ is a ToM problem and $Q$ is a query function that modifies $p$ according to a prompting technique, namely $Q:p\to p^{\prime }$.

背景符号。（大）语言模型是一个以先前输入或生成的 token 序列为条件，从有限词表中预测下一个 token 的函数，即 $\psi :\mathbf{v}\in V^{\ast }\to v\in V$。 通过以自回归方式调用 $\psi$，即 $\psi (v\mid \mathbf{v})$，可以使用这一机制连续采样多个 token 输出，直到预测出 end-of-text token。 在我们的设定中，问题被指定为元组 $(p,Q)$，其中 $p$ 是 ToM 问题，$Q$ 是按照提示技术修改 $p$ 的查询函数，即 $Q:p\to p^{\prime }$。

The LLM's output $y$ for an input $Q(p)$ is then compared for correctness against an oracle $\mathrm{\Omega }$, i.e., $\mathrm{\Omega }:\psi (Q(p))\to \{0,1\}$, where $1$ means correct classification ($0$, otherwise). On a sample of $N>0$ ToM problems, the accuracy of a model $\psi$ is then measured as $\frac{1}{N}\sum _{i=1}^N\mathrm{\Omega }(\psi (Q(p_i)))$, i.e., the average number of times a model is correct in its prediction.

随后，将 LLM 对输入 $Q(p)$ 的输出 $y$ 与判定器 $\mathrm{\Omega }$ 比较以判断正确性，即 $\mathrm{\Omega }:\psi (Q(p))\to \{0,1\}$，其中 $1$ 表示分类正确（否则为 $0$）。 在包含 $N>0$ 个 ToM 问题的样本上，模型 $\psi$ 的准确率被度量为 $\frac{1}{N}\sum _{i=1}^N\mathrm{\Omega }(\psi (Q(p_i)))$，即模型预测正确次数的平均值。

3.2.1 Discrete World Models via Prompting

Given a ToM problem $p$ and a constant $T\le |p|$, where $|p|$ is ideally measured as the number of state changes in the problem, we can rewrite $p$ as $p_1\oplus p_2\oplus \cdots \oplus p_T$. Our query function adds a standard preamble $x$ similar to that of CoT. DWM inserts, after each "split" $p_t$, an additional prompt $w$ like Now, provide a succinct description of the state of the environment and each agent's belief. and query an LLM to provide a representation of the current state description of the environment.

给定 ToM 问题 $p$ 和常数 $T\le |p|$，其中 $|p|$ 在理想情况下以问题中的状态变化次数衡量，我们可以将 $p$ 改写为 $p_1\oplus p_2\oplus \cdots \oplus p_T$。 我们的查询函数添加一个与 CoT 类似的标准前导语 $x$。 DWM 在每个“划分” $p_t$ 后插入额外提示 $w$，例如 Now, provide a succinct description of the state of the environment and each agent's belief.，并查询 LLM 以获得当前环境状态描述的表示。

An LLM is initially queried with $x\oplus p_1\oplus w$, and the answer $a_1$ is concatenated to the next query, i.e., $\psi (x\oplus p_1\oplus w\oplus a_1\oplus p_2\oplus w)$ to retrieve $a_2$. The process is carried on for each of the $T$ chunks, and, at the end, $y$ is concatenated to eventually prompt the model for the correct answer to $p$.

最初使用 $x\oplus p_1\oplus w$ 查询 LLM，再将答案 $a_1$ 拼接到下一次查询中，即通过 $\psi (x\oplus p_1\oplus w\oplus a_1\oplus p_2\oplus w)$ 检索 $a_2$。 对 $T$ 个片段逐一重复该过程，最后拼接 $y$，从而提示模型给出 $p$ 的正确答案。

Let $a_1=\psi (x\oplus p_1\oplus w)$, and for $t>1$ let:

令 $a_1=\psi (x\oplus p_1\oplus w)$，并对 $t>1$ 定义：

$$\begin{array}{rl}{a}_t& =\psi (x\oplus p_1\oplus w\oplus a_1\oplus p_2\oplus \cdots \oplus a_{t-1}\oplus p_t)\\ & =\psi (x\oplus (\underset{i=1}{\overset{t-1}{⨁}}{p}_i\oplus w\oplus a_i)\oplus p_t).\end{array}$$

Then, the final query is:

于是，最终查询为：

$$\psi (x\oplus (\underset{t=1}{\overset{T}{⨁}}{p}_t\oplus w\oplus a_t)\oplus y).$$

In this sense, our partition function (Definition 2) consists of splitting a prompt into sequential chunks of the prompt, while the LLM is prompted to provide each state event at time $1\le t<T$ as $e_t=\psi (x\oplus (\underset{i=1}{\overset{t}{⨁}}{p}_i\oplus w\oplus a_i)\oplus \omega )$. The process of prompting a model with DWM is illustrated in Figure 3.

在这一意义上，我们的划分函数（定义 2）把提示拆分为连续片段，同时提示 LLM 在 $1\le t<T$ 时以 $e_t=\psi (x\oplus (\underset{i=1}{\overset{t}{⨁}}{p}_i\oplus w\oplus a_i)\oplus \omega )$ 的形式给出每个状态事件。 使用 DWM 提示模型的过程如图3所示。

3.2.2 On the Complexity of DWM

DWM progressively calls an LLM $T>0$ times to generate informative states. For a ToM problem of length $n$ (i.e., the number of input tokens), which we assume, w.l.o.g., that can be split into $T$ chunks of approximately the same length $|x\oplus p_i\oplus w|=\frac{n}{T}$, the number of tokens generated by an LLM is of the order of $\mathcal{O}\left(\sum _{t=1}^T|x\oplus (\underset{i=1}{\overset{t-1}{⨁}}{p}_i\oplus w\oplus a_i)|\right)$, where $p_t$ ($a_t$) is the portion of the problem (answer) prompted (retrieved) at iteration $t$. With the further assumption that each answer retrieved at split $t\le T$ has the same length $o$, the complexity is further simplified to be asymptotic to $\mathcal{O}((\frac{n}{T}+o{)}^2)$.

DWM 逐步调用 LLM 共 $T>0$ 次，以生成有信息量的状态。 对于长度为 $n$（即输入 token 数量）的 ToM 问题，不失一般性地假设它可被划分为 $T$ 个长度近似相同的片段，满足 $|x\oplus p_i\oplus w|=\frac{n}{T}$；则 LLM 生成的 token 数量为 $\mathcal{O}\left(\sum _{t=1}^T|x\oplus (\underset{i=1}{\overset{t-1}{⨁}}{p}_i\oplus w\oplus a_i)|\right)$ 阶，其中 $p_t$（$a_t$）是第 $t$ 次迭代时提示（检索）的部分问题（答案）。 进一步假设在每个划分 $t\le T$ 处检索到的答案长度均为 $o$，则复杂度可进一步简化为渐近于 $\mathcal{O}((\frac{n}{T}+o{)}^2)$。

Compared to CoT, whose complexity is $\mathcal{O}(n+o)$, DWM requires an additional linear number of calls to the model. On the other hand, ToT with the same number of splits $\frac{n}{T}$ and $m>1$ experts results in even higher complexity, i.e., asymptotic to $\mathcal{O}(m(\frac{n}{T}+o{)}^2)$.

与复杂度为 $\mathcal{O}(n+o)$ 的 CoT 相比，DWM 需要额外线性数量的模型调用。 另一方面，在划分数量同为 $\frac{n}{T}$ 且有 $m>1$ 个专家时，ToT 会产生更高的复杂度，即渐近于 $\mathcal{O}(m(\frac{n}{T}+o{)}^2)$。

4. Experimental Evaluation

图4：GPT-3.5-Turbo（上）与 Mixtral 8x7B（下）在不同 ToM 任务上的结果，对比 DWM（一至五次划分）、CoT、ToT 和结构化提示（JSON 与 YAML）。

The experiments are organised as follows. We first test the performance of DWM on ToMi, MindGames, Adv-CSFB, SocialIQA, and FANToM, comparing it with CoT, ToT and prompting with structured data (struct), i.e., the model is queried to first represent the problem in a structured format such as JSON or Yaml. We further show that ToMi has been memorised word for word by GPT models, with CoT (and any technique that leaves the input unchanged) being the best-performing method. We then quantify the complexity of the benchmarks introduced above and highlight the correlation with the models' performances. Our framework shows complexity ranges between easy and hard problems, even within a benchmark. We conduct our experiments on GPT-3.5-Turbo, GPT-4, LLaMA3-70B and Mixtral 8x7B.

实验组织如下。 我们首先测试 DWM 在 ToMi、MindGames、Adv-CSFB、SocialIQA 和 FANToM 上的性能，并将其与 CoT、ToT 以及结构化数据提示（struct）比较；后者要求模型首先以 JSON 或 YAML 等结构化格式表示问题。 我们进一步表明，ToMi 已被 GPT 模型逐字记忆，因而 CoT（以及任何保持输入不变的技术）成为表现最佳的方法。 随后，我们量化上述基准的复杂度，并突出它与模型性能之间的相关性。 我们的框架表明，即使在同一基准内部，问题复杂度也覆盖从简单到困难的范围。 我们在 GPT-3.5-Turbo、GPT-4、LLaMA3-70B 和 Mixtral 8x7B 上开展实验。

表1：五个 ToM 基准的记忆测试汇总。我们提示 GPT-3.5-Instruct 预测随机抽取的 100 个测试样本的后续文本，并计算精确记忆率与模糊记忆率；下方同时列出不同模型中 CoT 与 DWM 的最佳性能。

	ToMi	FANToM	Mindgames	Adv-CSFB	SocialIQa
Memorisation - perfect match	52%	35%	2%	0%	0%
Memorisation - fuzzy	89 ± 15%	74 ± 24%	64 ± 18%	51 ± 11%	40 ± 12%
DWM	0.625	0.579	0.618	0.8364	0.691
CoT	0.629	0.403	0.552	0.7091	0.736

表2：不同 ToM 基准的状态性与无状态性汇总。底部给出能保证 GPT-3.5-Turbo 使用 DWM 时取得最高性能的划分数，它与各基准的状态性高度相关。

	ToMi	FANToM	Mindgames	Adv-CSFB	SocialIQa
Statefulness	2.62 ± 1.68	2.44 ± 0.96	1.22 ± 0.90	3.24 ± 1.35	1. ± 0.
Statelessness	4.27 ± 2.1	59.42 ± 18.91	5.24 ± 2.71	2.86 ± 1.34	1.14 ± 0.447
DWM - Best Split	3	3	1	4	1

4.1 DWM on ToM Benchmarks

图5：一个真实 ToMi 示例：GPT-4 使用 CoT 时失败，使用 DWM 时成功。CoT 引出了不真实的推理过程，而 DWM 用关于 Benjamin 一阶信念的隐式信息正确告知模型。

We report results for GPT-3.5-Turbo and Mixtral 8x7B on the five ToM benchmarks: for reasons of space, results for LLaMA3-8B, LLaMA3-70B and GPT-4 are reported in the appendix. As illustrated in Figure 4 (top), DWM improves the performance of GPT-3.5-Turbo on Mindgames, FANToM and Adv-CSFB by a solid margin. On SocialIQa, which has very short inputs, DWM performs slightly worse than CoT but better than ToT. On the other hand, on ToMi, the best prompting techniques are CoT and ToT.

我们报告 GPT-3.5-Turbo 和 Mixtral 8x7B 在五个 ToM 基准上的结果；由于篇幅限制，LLaMA3-8B、LLaMA3-70B 和 GPT-4 的结果见附录。 如图4顶部所示，DWM 显著提高了 GPT-3.5-Turbo 在 Mindgames、FANToM 和 Adv-CSFB 上的性能。 在输入很短的 SocialIQa 上，DWM 略逊于 CoT，但优于 ToT。 另一方面，在 ToMi 上，最佳提示技术是 CoT 和 ToT。

While one may think memorisation plays a role in boosting the performance of LLMs with these prompting techniques, in the next section, we provide evidence this hypothesis is not necessarily true. With Mixtral 8x7B (Figure 4, bottom), DWM improves the performance on ADVcsfb, FANToM, ToMi and Mindgames, and reaches that of CoT on SocialIQa.

人们可能认为记忆在这些提示技术提升 LLM 性能的过程中发挥了作用，但下一节将给出证据，说明该假设不一定成立。 在 Mixtral 8x7B 上（图4底部），DWM 提升了 ADVcsfb、FANToM、ToMi 和 Mindgames 上的性能，并在 SocialIQa 上达到 CoT 的水平。

DWM elicits more informed state spaces. We qualitatively analysed the information elicited by an LLM when prompted with DWM and discovered that it forces a model to output information not explicitly available in the prompt. Consider the ToMi example in Figure 5 where GPT-4 is prompted with a situation where agents interact and are then queried with the first-order belief of Benjamin. With CoT, the model makes an erroneous assumption about the presence of Benjamin and Isabella in the room. On the other hand, when prompted with DWM, GPT-4 provides an informative description of each state space, particularly the knowledge and the uncertainty of each agent's beliefs, and eventually answers correctly.

DWM 引出信息更丰富的状态空间。我们对使用 DWM 提示 LLM 时引出的信息进行了定性分析，发现它会迫使模型输出提示中未显式提供的信息。 考虑图5中的 ToMi 示例，其中 GPT-4 接收到一个智能体交互场景，随后被询问 Benjamin 的一阶信念。 使用 CoT 时，模型错误地假设 Benjamin 和 Isabella 是否在房间中。 另一方面，使用 DWM 提示时，GPT-4 会给出每个状态空间的信息性描述，尤其是每个智能体信念中的知识与不确定性，并最终正确作答。

One example per benchmark is available in the appendix, while many more are available for inspection in the Code Supplementary Material. This phenomenon is ubiquitous to all the ToM tasks we tested, a hint that DWM may elicit the ToM capabilities of LLMs without requiring external information or solvers.

每个基准的一个示例见附录，代码补充材料中还提供了更多可供检查的示例。 这一现象普遍存在于我们测试的所有 ToM 任务中，暗示 DWM 可能无需外部信息或求解器即可引出 LLM 的 ToM 能力。

图6：五个 ToM 基准按复杂度升序排列的箱线图，并报告 GPT-3.5-Turbo、GPT-4、Mixtral 8x7B 和 LLaMA3-70B 使用 CoT 时的平均错误率。

图7：五个 ToM 基准的状态性（左）、无状态性（中，纵轴为对数尺度）与复杂度（右）分析。图中同时给出均值、标准差、离群点和最佳 DWM 划分数，划分数与状态性呈强相关。

Memorisation in Theory of Mind. Recent works expressed concern about ToM benchmarks' efficacy in memorisation. This motivated us to quantify and then analyse the impact of memorisation of ToM benchmarks on performance. We computed the percentage of memorised prompts to understand whether that affects the performance of techniques, such as DWM, that split the prompt into chunks and introduce additional information instead of CoT, which leaves the input prompt unchanged.

**心智理论中的记忆。**近期工作对 ToM 基准受记忆影响后的有效性表示担忧。 这促使我们量化并分析 ToM 基准记忆现象对性能的影响。 我们计算被记忆提示的比例，以了解这是否会影响 DWM 等技术的性能；这些技术会把提示拆分成片段并引入额外信息，而 CoT 则保持输入提示不变。

As illustrated in Table 1, ToMi and FANToM have been heavily memorised, with entire portions of the benchmarks that can be retrieved word for word from GPT-3.5-Instruct (the autocomplete model by OpenAI). Despite that, no clear evidence of a performance drop in DWM induced by memorisation exists. For GPT-3.5, despite CoT having higher performance on ToMi, DWM is better on FANToM (Figure 4).

如表1所示，ToMi 和 FANToM 已被大量记忆，基准中的整段内容都可以从 GPT-3.5-Instruct（OpenAI 的自动补全模型）中逐字检索出来。 尽管如此，并没有清晰证据表明记忆导致 DWM 性能下降。 对于 GPT-3.5，尽管 CoT 在 ToMi 上表现更好，但 DWM 在 FANToM 上更优（图4）。

We hypothesise that as long as a memorised problem is prompted, either in its exact form (as for CoT) or split as in DWM, the most potent models can recover it alongside the ground truth label, thus invalidating the test for both. We conclude with a note of caution: while we discovered that ToMi and FANToM are memorised by GPT-3.5-Instruct, that does not imply any LLM, including GPT-3.5-Turbo and GPT-4, whose training details are not released publicly, has been trained on that data.

我们假设，只要被记忆的问题被提示，无论以完整形式输入（如 CoT）还是像 DWM 那样拆分输入，能力最强的模型都可以将问题连同真实标签一起恢复，从而使两种测试都失效。 最后需要谨慎说明：尽管我们发现 ToMi 和 FANToM 被 GPT-3.5-Instruct 记忆，但这并不意味着任何 LLM，包括训练细节未公开的 GPT-3.5-Turbo 和 GPT-4，都曾在这些数据上训练。

4.2 Statefulness of ToM Benchmarks

We used the complexity framework introduced in Section 3.1 to characterise the statefulness and statelessness of the five ToM benchmarks used for the experimental evaluation. We randomly sampled $50$ problems from each dataset, identified the objects, and manually labelled stateful and stateless state events. We release the split samples alongside a web application that facilitates manual labelling. As illustrated in Figure 7 (left), the statefulness of each problem, i.e., that of the object a model must track to answer correctly, strongly correlates with the best-performing DWM split.

我们使用第 3.1 节引入的复杂度框架，刻画实验评估所用五个 ToM 基准的状态性与无状态性。 我们从每个数据集中随机抽取 $50$ 个问题，识别其中的对象，并人工标注有状态和无状态的状态事件。 我们发布这些划分样本，同时提供一个便于人工标注的网页应用。 如图7左侧所示，每个问题的状态性，即模型为了正确回答而必须追踪的对象状态，与表现最佳的 DWM 划分数高度相关。

The statelessness complexity, reported in Figure 7 (middle), i.e., that of objects that a model does not need to track, grows larger for problems such as FANToM, only partially influencing the models' performance. We hypothesise that the most potent models developed some competency in discerning the relevant part of a prompt (the stateful events) from the confounding ones. We finally report, in Figure 7 (right), the complexity of each problem computed as per the complexity equation, with $\tau$ set in a range between $0.05$ and $0.2$ (i.e., the relative weight of stateless compared to stateful events). Results suggest that FANToM is the most difficult ToM task for humans and LLMs (see Figure 4), followed by ToMi (the second most difficult for LLMs as well) and Adv-CSFB (which seems easier than the others); in contrast, Mindgames and SocialIQa tend to be easier.

无状态复杂度见图7中部，即模型无需追踪的对象复杂度；它在 FANToM 等问题上更大，但只会部分影响模型性能。 我们假设，能力最强的模型已经形成了某种能力，可以将提示中的相关部分（有状态事件）与混淆部分区分开来。 最后，我们在图7右侧报告按照复杂度公式计算的每个问题的复杂度，其中 $\tau$ 设置在 $0.05$ 到 $0.2$ 之间（即无状态事件相对于有状态事件的权重）。 结果表明，FANToM 是对人类和 LLM 而言最困难的 ToM 任务（见图4），其次是 ToMi（对 LLM 而言同样是第二困难）和 Adv-CSFB（似乎比其他任务更容易）；相比之下，Mindgames 和 SocialIQa 往往更简单。

Finally, in Figure 6, we compare the accuracy of GPT-3.5-Turbo, GPT-4, Mixtral 8x7B and LLaMA3-70B when prompted with CoT (i.e., without split) on the five ToM benchmarks with the complexity of the task as per the complexity definition. We observe a strong correlation between the error-rate and the complexity of a task, i.e., our framework correctly identifies the tasks that are harder both for humans and current state-of-the-art LLMs.

最后，在图6中，我们比较 GPT-3.5-Turbo、GPT-4、Mixtral 8x7B 和 LLaMA3-70B 使用 CoT（即不划分）时在五个 ToM 基准上的准确率，以及由复杂度定义得到的任务复杂度。 我们观察到错误率与任务复杂度之间存在强相关，也就是说，我们的框架正确识别了对人类和当前最先进 LLM 都更困难的任务。

5. Conclusions

This paper introduces a complexity framework to measure the difficulty of Theory of Mind (ToM) problems. It quantifies the difficulty by tracking necessary states (stateful) and unnecessary states (stateless), with the latter discounted in the complexity computation. The framework evidences a strong correlation between complexity and model performance. Inspired by this framework, we propose DWM, a prompting technique that splits a prompt into parts to query a model for a consistent representation of the environment and agents' beliefs. DWM outperforms CoT and ToT by extracting implicit but relevant information.

本文引入一个复杂度框架，用于衡量心智理论（ToM）问题的难度。 它通过追踪必要状态（有状态）和非必要状态（无状态）来量化难度，并在复杂度计算中对后者进行折扣。 该框架表明复杂度与模型性能之间存在强相关。 受这一框架启发，我们提出 DWM，这种提示技术将提示拆分成多个部分，查询模型以获得环境和智能体信念的一致表示。 DWM 通过提取隐式但相关的信息而优于 CoT 和 ToT。

Limitations

Higher order belief tracking. Our theoretical framework reduces the problem of solving a belief ToM problem to finding the correct descriptions that need to be tracked. It extends seamlessly to tasks with much higher complexity, however, we have not had the opportunity to test this theory in those settings. We noticed that most theory of mind tasks available in the community only require one to five states to be correctly answered. A possible extension would be testing the theory upon tasks with higher state complexity, e.g. $k^{\text{th}}$-order belief tracking tasks. However, it is unclear whether this could be useful in real applications as most human belief tracking is limited to 5 or 6 orders.

**高阶信念追踪。**我们的理论框架将求解信念类 ToM 问题归结为寻找需要追踪的正确描述。 它可以无缝扩展到复杂度高得多的任务，但我们尚未有机会在这些设定中检验该理论。 我们注意到，社区中现有的大多数心智理论任务只需追踪一至五个状态即可正确回答。 一种可能的扩展是在状态复杂度更高的任务上检验该理论，例如 $k^{\text{th}}$ 阶信念追踪任务。 然而，这在实际应用中是否有用尚不明确，因为大多数人类信念追踪仅限于 5 或 6 阶。

On task splitting methods. It is not straightforward to automatically find the correct task splits in a manner that correctly describes the state. An LLM could find a way to split it by itself correctly and use those splits to answer the question. We attempted this approach, yet with a simple prompting method, the model splits every sentence, making the descriptions much noisier and less accurate. Future work could try to find the best splits automatically.

**关于任务划分方法。**自动找到能够正确描述状态的任务划分并不直接。 LLM 可以设法自行正确划分任务，并使用这些划分回答问题。 我们尝试了这种方法，但使用简单提示时，模型会拆分每个句子，使描述噪声更大且准确度更低。 未来工作可以尝试自动寻找最佳划分。

Memorization analysis. Training and evaluating on the same dataset produces positively biased data on the model's performance. While running our benchmarks on ToMi, we discovered that the GPT-3.5 model had completely memorized parts of the dataset. This motivated us to extend the memorization test to the other tasks. We urge the research community to include a memorization section on every benchmark study with public datasets used in their works. This data is crucial to conduct fair and unbiased research on evaluating LLMs' abilities. Future works will include an analysis of the memorisation rate of other ToM tasks alongside tests to quantify their impact on different models.

**记忆分析。**在同一数据集上训练和评估会使模型性能数据产生正向偏差。 在 ToMi 上运行基准时，我们发现 GPT-3.5 模型已经完全记住了数据集的一部分。 这促使我们将记忆测试扩展到其他任务。 我们呼吁研究社区在所有使用公开数据集的基准研究中加入记忆分析章节。 这些数据对于公平、无偏地评估 LLM 能力至关重要。 未来工作将分析其他 ToM 任务的记忆率，并通过测试量化它们对不同模型的影响。

On element interactivity. Sweller proposes a measure of complexity for cognitive tasks that encompasses three main components, namely the intrinsic, extraneous, and germane cognitive load. In its framework, which has wide applications in education, the intrinsic load relates to the number of references, or interactions, between the elements of a problem, i.e., the information or concept that needs to be understood to answer the question. Our framework approximates the intrinsic and extraneous loads to be single sentences in a ToM problem, which is not assured to be the best measure.

**关于要素交互性。**Sweller 提出一种认知任务复杂度度量，包含三个主要组成部分，即内在、外在和相关认知负荷。 在这一广泛应用于教育领域的框架中，内在负荷与问题要素之间的引用或交互数量有关，也就是回答问题时需要理解的信息或概念。 我们的框架将内在和外在负荷近似为 ToM 问题中的单个句子，但这未必是最佳度量。

Ethical Statement

The datasets and pre-trained LLMs that we use are all publicly available. This paper focuses on ToM problems' hardness and prompting methods. We highlight that LLMs do not guarantee the production of factual data or correct reasoning steps, and the prompting methods developed here should not be regarded as the source of truth in decision-making.

我们使用的数据集和预训练 LLM 均公开可用。 本文聚焦 ToM 问题的难度和提示方法。 我们强调，LLM 无法保证生成事实性数据或正确推理步骤，本文开发的提示方法不应被视为决策中的唯一事实来源。