---

Human-inspired Episodic Memory for Infinite Context LLMs

MemoryLong Context270+ICLR 2025CCF-A华为诺亚方舟实验室UCL

Fountas Z, Benfeghoul M A, Oomerjee A, et al. Human-inspired Episodic Memory for Infinite Context LLMs. ICLR, 2025.

https://openreview.net/forum?id=BI2int5SAC

https://github.com/em-llm/EM-LLM-model

---

面向无限上下文 LLM 的人类启发式情景记忆

Abstract

Large language models (LLMs) have shown remarkable capabilities, but still struggle with processing extensive contexts, limiting their ability to maintain coherence and accuracy over long sequences. In contrast, the human brain excels at organising and retrieving episodic experiences across vast temporal scales, spanning a lifetime. In this work, we introduce EM-LLM, a novel approach that integrates key aspects of human episodic memory and event cognition into LLMs with no fine-tuning, enabling them to handle practically infinite context lengths while maintaining computational efficiency. EM-LLM organises sequences of tokens into coherent episodic events using a combination of Bayesian surprise and graph-theoretic boundary refinement in an online fashion. When needed, these events are retrieved through a two-stage memory process, combining similarity-based and temporally contiguous retrieval for efficient, human-inspired access to relevant information. Experiments on the LongBench and $\mathrm{\infty }$-Bench benchmarks demonstrate EM-LLM's superior performance, consistently outperforming the state-of-the-art retrieval model InfLLM across various baseline LLMs. In addition, EM-LLM outperforms its popular counterpart, RAG, in a wide range of tasks, while requiring similar resources. Notably, EM-LLM's performance even surpasses full-context models in most tasks, while successfully performing retrieval across 10 million tokens -- a scale computationally infeasible for such models. Finally, our analysis reveals strong correlations between EM-LLM's event segmentation and human-perceived events, suggesting parallels between this artificial system and its biological counterpart, thereby offering a novel computational framework for exploring human memory mechanisms.

大语言模型（LLM）已经展现出卓越能力，但仍难以处理超长上下文，因而限制了它们在长序列中维持连贯性和准确性的能力。 相比之下，人脑擅长在跨越一生的巨大时间尺度上组织和检索情景经历。 在这项工作中，我们提出 EM-LLM，这是一种无需微调即可把人类情景记忆和事件认知的关键方面融入 LLM 的新方法，使其能够处理几乎无限的上下文长度，同时保持计算效率。 EM-LLM 结合贝叶斯惊奇度与图论边界细化，以在线方式把 token 序列组织成连贯的情景事件。 需要时，系统通过两阶段记忆过程检索这些事件，结合基于相似度和时间连续性的检索，以高效且受人类启发的方式访问相关信息。 在 LongBench 和 $\mathrm{\infty }$-Bench 基准上的实验表明，EM-LLM 具有更优性能，在多种基础 LLM 上持续优于最先进的检索模型 InfLLM。 此外，EM-LLM 在广泛任务上优于其常用对应方法 RAG，同时所需资源相近。 值得注意的是，EM-LLM 在多数任务上的表现甚至超过完整上下文模型，并成功跨越 1000 万个 token 进行检索，而这样的规模对完整上下文模型而言在计算上不可行。 最后，我们的分析揭示了 EM-LLM 事件分割与人类感知事件之间的强相关性，表明这一人工系统与其生物学对应物之间存在相似之处，并由此为探索人类记忆机制提供了一个新的计算框架。

1. Introduction

For contemporary pre-trained large language models (LLMs), the context window serves as the primary mechanism to incorporate domain-specific, private, or common up-to-date information. However, despite their remarkable and ever-expanding capabilities, LLMs still exhibit significant limitations when tasked with processing extensive contexts. These limitations stem from inherent challenges in Transformer-based architectures. Recent studies have shown that Transformers struggle with extrapolating to contexts longer than their training window size. On top of this, employing softmax attention over extended token sequences requires substantial computational resources for each token generation, while the resulting aggregated embeddings (the weighted sums of value vectors) risk becoming excessively noisy and losing their distinctiveness.

对于当代预训练大语言模型（LLM），上下文窗口是引入特定领域、私有或常见最新信息的主要机制。 然而，尽管能力卓越且不断扩展，LLM 在处理超长上下文时仍表现出明显局限。 这些局限源于 Transformer 架构的固有挑战。 近期研究表明，Transformer 难以外推到比训练窗口更长的上下文。 此外，在扩展的 token 序列上使用 softmax 注意力，会使每次 token 生成都需要大量计算资源，而得到的聚合嵌入（值向量的加权和）还可能变得过于嘈杂并失去区分度。

To mitigate these challenges, recent works have focused on retrieval-based methods, either in the form of in-context augmentation (e.g., retrieval-augmented generation (RAG)-based techniques) or via retrieval of previously-inferred key-value pairs (KV) within individual attention heads. Notably, state-of-the-art (SOTA) performance is achieved when KV pairs are initially organised into non-overlapping segments and then retrieved together as one block of sequential tokens. While such techniques present interesting research avenues, we still see a significant gap between the performance of LLMs in short- vs long-context tasks, even when existing long-context architectures are employed.

为缓解这些挑战，近期工作聚焦于基于检索的方法，包括上下文内增强（例如基于检索增强生成（RAG）的技术），或在各个注意力头中检索此前推断出的键值对（KV）。 值得注意的是，先把 KV 对组织成互不重叠的分段，再把它们作为一块连续 token 共同检索，可以取得当前最先进的性能。 尽管这些技术提供了有趣的研究方向，但即使采用现有长上下文架构，我们仍然观察到 LLM 在短上下文任务与长上下文任务之间存在显著性能差距。

图1：上：以 LLaMA-3.1-8B 为基础 LLM，在 LongBench 上比较 EM-LLM_S（仅惊奇度）、RAG（NV-Embed-v2 检索器）与完整上下文。下：在扩展版 ∞-Bench Retrieve.PassKey 上比较不同长序列方法，按上下文窗口长度排序；基线数据取自 Ding 等人。

This work tackles the above challenges and attempts to bridge this performance gap by taking inspiration from the algorithmic interpretation of episodic memory in the human brain -- the memory system responsible for encoding, storing, and retrieving personal experiences and events. The brain makes sense of its continuous experience in the real world by segmenting it into discrete episodic events, which are first organised in a hierarchical and nested-timescale structure and then stored in long-term memory. Notably, the boundaries between such events are the access points for memory retrieval and are widely believed to correspond to points in time with high prediction errors between the brain's generative model and its raw sensory input (a.k.a., surprise). In this context, surprise refers to moments when the brain's predictions about incoming sensory information are significantly violated, leading to a mismatch between what is expected and what is actually perceived. These instances of high surprise are thought to signal important changes in the environment or narrative, prompting the brain to segment the ongoing experience into distinct events. Once segmented and stored, the brain recalls episodic memories based on their similarity to current experience, recency, original temporal order, and their proximity to other recalled memories (temporal asymmetry and contiguity).

这项工作通过借鉴对人脑情景记忆的算法解释来应对上述挑战并尝试弥合这一性能差距；情景记忆是负责编码、存储和检索个人经历与事件的记忆系统。 大脑通过把现实世界中的连续经历分割成离散情景事件来理解它们，这些事件先被组织成具有层次结构和嵌套时间尺度的形式，再存入长期记忆。 值得注意的是，这些事件之间的边界是记忆检索的访问点，并且被广泛认为对应于大脑生成模型与原始感官输入之间预测误差较高的时间点，即惊奇度。 在这一语境下，惊奇度是指大脑对传入感官信息的预测被显著违背，导致预期内容与实际感知内容不匹配的时刻。 这些高惊奇度时刻被认为预示着环境或叙事中的重要变化，从而促使大脑把持续经历分割成不同事件。 完成分割和存储后，大脑会依据情景记忆与当前经历的相似度、近因性、原始时间顺序，以及它们与其他已回忆记忆的邻近程度（时间不对称性和连续性）来进行回忆。

Contributions: We propose EM-LLM, a novel architecture integrating crucial aspects of event cognition and episodic memory into Transformer-based LLMs through three key innovations (a, b and c). For memory formation, we segment input token sequences into memory units representing episodic events. The boundaries of these units are (a) initially determined using the model's surprise level during inference, then (b) refined to maximize within-unit cohesion and cross-unit separation (see Section 3.2). This refinement leverages graph-theoretic metrics, treating attention key similarity as a weighted adjacency matrix, and aims to enhance efficient information recall in complex, long-context tasks: by consolidating related information into single units, we seek to minimize the number of units needed for event-specific recall. The resulting memory formation process is computationally efficient: surprise-based segmentation requires no additional computation, and refinement complexity is $\mathcal{O}(nm)$, where $m$ is typically negligible compared to the token count $n$ in long-context tasks. For memory recall, (c) our approach combines similarity-based retrieval with temporal contiguity and asymmetry mechanisms, building on recently discovered parallels between LLMs and human sequential information retrieval patterns. This method therefore ensures efficient information access while replicating temporal dynamics from human free recall studies, and enhancing performance on tasks requiring temporal reasoning. See the appendix for analysis of EM-LLM's architectural contributions.

贡献： 我们提出 EM-LLM，这是一种通过三项关键创新（a、b 和 c）把事件认知与情景记忆的重要方面融入 Transformer LLM 的新架构。 在记忆形成阶段，我们把输入 token 序列分割成表示情景事件的记忆单元。 这些单元的边界会（a）先利用推理期间的模型惊奇度确定，再（b）通过细化最大化单元内聚合度和单元间分离度（见第3.2节）。 这种细化利用图论指标，把注意力键相似度视为加权邻接矩阵，旨在提高复杂长上下文任务中的信息回忆效率：通过把相关信息整合到单个单元中，我们力求最小化针对特定事件进行回忆所需的单元数量。 由此得到的记忆形成过程具有计算效率：基于惊奇度的分割不需要额外计算，细化复杂度为 $\mathcal{O}(nm)$，其中在长上下文任务中，$m$ 相比 token 数量 $n$ 通常可以忽略。 在记忆回忆阶段，（c）我们的方法结合基于相似度的检索、时间连续性和不对称性机制，并建立在近期发现的 LLM 与人类序列信息检索模式之间的相似性之上。 因此，该方法在复现人类自由回忆研究中的时间动态的同时，确保了高效的信息访问，并增强了需要时间推理的任务表现。 附录分析了 EM-LLM 的架构贡献。

Performance: We show that our method is scalable and significantly outperforms the SOTA retrieval model InfLLM, as well as RAG and full-context methods, on the widely-used LongBench and $\mathrm{\infty }$-Bench benchmarks designed for long-context tasks (see Figure 1). Furthermore, we perform successful passkey retrieval across $10$M tokens, a length which is computationally infeasible for current full-context models. To further prove our hypotheses, we then employ a series of human-annotated podcast scripts to show that information in LLM attention heads can be semantically grouped in a way that correlates with the event structure perceived by humans. Therefore, LLM-perceived surprise can indeed serve as a proxy for the cognitive signals that drive human event segmentation, as confirmed by previous studies. Finally, using the long-context PG-19 dataset, which comprises a diverse corpus of English books, we evaluate the effectiveness of our segmentation method for grouping relevant information and assess the performance of different boundary refinement objectives.

性能： 我们表明，该方法具有可扩展性，并且在广泛使用、专为长上下文任务设计的 LongBench 和 $\mathrm{\infty }$-Bench 基准上显著优于最先进的检索模型 InfLLM、RAG 和完整上下文方法（见图1）。 此外，我们成功跨越 $10$M 个 token 完成密钥检索，而这样的长度对当前完整上下文模型而言在计算上不可行。 为进一步验证我们的假设，我们随后使用一系列经人工标注的播客脚本，表明 LLM 注意力头中的信息能够以与人类感知事件结构相关的方式进行语义分组。 因此，正如此前研究所证实的，LLM 感知到的惊奇度确实可以作为驱动人类事件分割的认知信号的替代指标。 最后，我们使用包含多样化英文书籍语料的长上下文 PG-19 数据集，评估分割方法对相关信息进行分组的有效性，并评估不同边界细化目标的性能。

2. Related Work

2.1 Long-context in LLMs

Recently, several approaches have been proposed to extend the context window of Transformer-based models. These include methods that address the limited representational capacity of softmax attention, and its quadratic computational and memory cost. Other methods target the poor extrapolation of typical positional encodings to out-of-distribution context lengths. The latter is evident in most widely used methods, including the original absolute positional encodings and the more recent relative positional encodings, such as the Rotary Positional Embeddings (RoPE). To address this, some methods propose scaling of the rotation angles or the base constant in RoPE. Others, scale positions without affecting the embedding function, explore alternative strategies such as KERPLE and FIRE or adopt relative position mechanisms from certain LMs like T5.

近期，人们提出了多种扩展 Transformer 模型上下文窗口的方法。 其中包括解决 softmax 注意力表示容量有限，以及其计算和内存成本呈二次增长的方法。 其他方法则针对典型位置编码无法良好外推到分布外上下文长度的问题。 后一个问题在大多数广泛使用的方法中都很明显，包括最初的绝对位置编码，以及旋转位置嵌入（RoPE）等较新的相对位置编码。 为解决这一问题，一些方法提出缩放 RoPE 的旋转角或基数常量。 其他方法则在不影响嵌入函数的情况下缩放位置、探索 KERPLE 和 FIRE 等替代策略，或采用 T5 等语言模型中的相对位置机制。

Concerning computational efficiency and diluted attention, successful approaches propose methods for general improvements to Transformer efficiency through optimised computations or compression techniques, as well as training methods tailored for long-context scenarios. Another direction is the utilisation of retrieval-based methods, the vast majority of which relies on a vector database that keeps a key-value cache and scalable approximations of k-nearest neighbors (k-NNs) to perform lookups. Interestingly, since using a key-value cache with k-NN lookup can be seen as an approximation of applying softmax attention to the full token sequence, k-NN retrieval methods can be used without fine-tuning. For an exception that does not rely on k-NNs, see Wang et al.

针对计算效率和注意力稀释问题，成功的方法通过优化计算或压缩技术普遍提高 Transformer 效率，也有方法采用针对长上下文场景定制的训练方式。 另一个方向是使用基于检索的方法，其中绝大多数依赖维护键值缓存的向量数据库，并使用可扩展的近似 k 近邻（k-NN）进行查找。 有趣的是，由于采用带 k-NN 查找的键值缓存可以视为对完整 token 序列应用 softmax 注意力的近似，因此 k-NN 检索方法无需微调即可使用。 不依赖 k-NN 的例外可参见 Wang 等人的工作。

A recent and interesting variant of k-NN retrieval involves retrieving large groups of tokens, rather than individual ones. Models that rely on this approach include SLED and the more recent InfLLM, which achieves SOTA performance on long-context benchmarks. InfLLM segments the entire context length into fixed-size memory units and employs k-NN lookup using the tokens with the highest accumulated scores per unit. The latter can be seen as a form of hierarchical attention in models that use such retrieval, as illustrated in Figure 2. While group-based retrieval represents a promising direction, our approach significantly advances this concept by dynamically determining token groupings in a manner akin to human memory formation. This effectively addresses a fundamental limitation of InfLLM's fixed-size segmentation and enables more adaptive and context-sensitive processing of extended information.

k-NN 检索近期出现了一个有趣变体：检索大组 token，而不是单个 token。 依赖这种方法的模型包括 SLED 和更近期的 InfLLM，后者在长上下文基准上取得了最先进的性能。 InfLLM 把整个上下文长度分割成固定大小的记忆单元，并使用每个单元中累积分数最高的 token 执行 k-NN 查找。 在使用这种检索的模型中，后一种做法可以视为一种层次注意力，如图2所示。 尽管基于分组的检索代表了一个有前景的方向，我们的方法通过以类似人类记忆形成的方式动态确定 token 分组，显著推进了这一概念。 这有效解决了 InfLLM 固定大小分割的根本局限，并使扩展信息的处理更具适应性和上下文敏感性。

图2：基于分组的 k-NN 检索可以视为一种层次情景注意力。系统先选择 k = 4 组 token（左），再将其用于 softmax 注意力（右），效果如同把其他相似度分数全部置零。该框架可以支持多层情景注意力。

2.2 Neural models of Episodic Memory and Event Cognition

The concept of episodic memory, central to our approach, has been extensively studied in both theoretical neuroscience and machine learning. Neural models of episodic memory capture human behaviour and neuroimaging data, providing insights into how the brain processes and stores experiences and suggesting links between memory, efficient representations and navigation of physical and conceptual spaces. In machine learning, episodic memory-inspired approaches have yielded significant improvements across various domains. For instance, episodic control has enhanced reinforcement learning agents' performance and learning speed. In addition, models of memory construction and consolidation have been successful in alleviating catastrophic forgetting in neural networks, including LLMs, and appear to explain key features of human memory, such as imagination and future thinking.

情景记忆是我们方法的核心概念，在理论神经科学和机器学习中都得到了广泛研究。 情景记忆的神经模型能够捕捉人类行为和神经成像数据，帮助我们理解大脑如何处理和存储经历，并表明记忆、高效表示以及对物理空间和概念空间的导航之间存在联系。 在机器学习中，受情景记忆启发的方法已经在多个领域带来显著改进。 例如，情景控制提高了强化学习智能体的性能和学习速度。 此外，记忆构建和巩固模型已经成功缓解了包括 LLM 在内的神经网络灾难性遗忘，并且似乎能够解释想象和未来思考等人类记忆的关键特征。

These models have revealed key aspects of episodic memory, particularly in describing how experiences are segmented into events, and when new memories are encoded and retrieved. Surprise plays a critical role in this process, triggering event boundaries and memory formation. This event-based structure is deeply intertwined with our perception of time, highlighting the interdependence of memory and temporal cognition. This insight has helped generative models for video and reinforcement learning to capture temporal dynamics more accurately. In terms of memory retrieval, studies in human free recall have shown a distinctive increased likelihood of retrieving items encoded close together in time (temporal contiguity) and in succession (temporal asymmetry) (see Figure 3A). Recently, it was shown that attention heads in Transformer-based LLMs that are associated with in-context learning, already exhibit the same dynamic retrieval behaviour (Figure 3B) including both contiguity and asymmetry effects. Therefore, Transformers have the inherent ability to act as episodic memory retrieval models, if provided with the right information within their context window. Our work leverages these concepts of surprise-based event segmentation and LLMs' inherent temporal contiguity and asymmetry effects to enable a new generation of Infinite Context-Length LLMs, capable of processing and understanding information over vastly extended timescales.

这些模型揭示了情景记忆的关键方面，尤其是经历如何被分割成事件，以及新记忆何时被编码和检索。 惊奇度在这一过程中发挥关键作用，会触发事件边界和记忆形成。 这种基于事件的结构与我们的时间感知深度交织，突出了记忆与时间认知的相互依赖。 这一见解帮助视频生成模型和强化学习更准确地捕捉时间动态。 在记忆检索方面，人类自由回忆研究表明，人们更有可能共同检索在时间上相邻编码的项目（时间连续性），并按先后顺序检索它们（时间不对称性）（见图3A）。 近期研究表明，Transformer LLM 中与上下文学习相关的注意力头已经表现出相同的动态检索行为（图3B），其中同时包括连续性和不对称性效应。 因此，只要在上下文窗口中提供恰当信息，Transformer 就天然具备充当情景记忆检索模型的能力。 我们的工作利用基于惊奇度的事件分割概念，以及 LLM 固有的时间连续性与不对称性效应，构建新一代无限上下文长度 LLM，使其能够在大幅扩展的时间尺度上处理和理解信息。

图3：（A）人类自由回忆中的时间连续性与不对称性效应示例。（B）GPT2 注意力头在所有测试 token 上的平均注意力分数。（C）每层中的记忆形成与检索流程：基于惊奇度分割输入、形成情景记忆、通过 k-NN 检索连续事件，并组成包含初始 token、连续性缓冲区、相似度缓冲区和局部上下文的最终上下文窗口。

3. EM-LLM: LLM with Episodic Memory

3.1 Architecture

EM-LLM is designed to be applied directly to pre-trained LLMs, enabling them to handle context lengths significantly larger than their original training length. Our architecture, illustrated in Figure 3C, divides the context into three distinct groups: initial tokens, evicted tokens and local context. This structure, while incorporating insights from recent work on token block retrieval, introduces novel elements inspired by human episodic memory.

EM-LLM 被设计为直接应用于预训练 LLM，使其能够处理远大于原始训练长度的上下文。 如图3C 所示，我们的架构把上下文分成三个不同部分：初始 token、被逐出的 token 和局部上下文。 这一结构在吸收近期 token 块检索工作见解的同时，引入了受人类情景记忆启发的新元素。

The local context represents the most recent tokens, maximising information about the current task, and fits within the typical context window of the underlying LLM. This group utilises full softmax attention and plays a role similar to the focus of attention in cognitive models of working memory, holding the most immediately relevant information for the current task. The evicted tokens typically comprise the majority of past tokens in a long-context scenario, extending far beyond the LLM's original training length. These tokens are managed by our proposed memory model functioning similarly to short-term episodic memory in the brain. Finally, following previous work, we also maintain a group of $128$ initial tokens in the LLM context. These act as attention sinks and help recover the performance of window attention, as first observed by Xiao et al. and Han et al. and later adopted by Xiao et al. For retrieved tokens, which are therefore discontinuous and outside the local context, we assign a fixed position embedding as in Raffel et al. and Xiao et al. This architecture enables EM-LLM to effectively process and utilise information from positions outside its pre-trained local context window, while maintaining the underlying LLM's performance characteristics.

局部上下文表示最近的 token，最大程度保留当前任务的信息，并且位于基础 LLM 的典型上下文窗口内。 这一部分使用完整 softmax 注意力，其作用类似于工作记忆认知模型中的注意焦点，保存当前任务最直接相关的信息。 在长上下文场景中，被逐出的 token 通常构成大多数历史 token，其范围远远超出 LLM 的原始训练长度。 这些 token 由我们提出的记忆模型管理，其功能类似于大脑中的短期情景记忆。 最后，沿用此前工作，我们还在 LLM 上下文中保留一组 $128$ 个初始 token。 这些 token 充当注意力汇点并帮助恢复窗口注意力的性能，这一现象最早由 Xiao 等人和 Han 等人观察到，随后由 Xiao 等人采用。 对于不连续且位于局部上下文之外的检索 token，我们按照 Raffel 等人和 Xiao 等人的方法为其分配固定位置嵌入。 这一架构使 EM-LLM 能够有效处理和利用预训练局部上下文窗口以外位置的信息，同时保持基础 LLM 的性能特征。

3.2 Memory formation via Surprise

In the context of LLMs, we define episodic memory as the organised, event-based collection of past key-value pairs, analogous to the latent representations of personal experiences in human memory. Just as unexpected or novel information plays a crucial role in human memory formation, we posit that analogous indicators of novelty in LLMs can serve as an effective proxy for identifying significant ``events'' within the model's experience. In Bayesian terms, surprise is quantified by the negative log-likelihood of observing the current, ground-truth token given the previous tokens in an auto-regressive model, with high values indicating the unpredictability or novelty of each new token within the context according to the model, i.e., being ``surprised'' by the next token.

在 LLM 语境下，我们把情景记忆定义为以事件为基础、有组织的历史键值对集合，类似于人类记忆中个人经历的潜在表示。 正如意外或新颖信息在人类记忆形成中发挥关键作用一样，我们认为 LLM 中类似的新颖性指标可以作为有效替代指标，用于识别模型经历中的重要“事件”。 在贝叶斯意义上，惊奇度由自回归模型在给定先前 token 时观察到当前真实 token 的负对数似然来量化；高值表示根据模型判断，每个新 token 在上下文中具有较高不可预测性或新颖性，即模型对下一个 token 感到“惊讶”。

Following work on cognitive modelling, we employ a thresholding mechanism to perform an initial identification of event boundaries (used for the first time in LLMs). Formally, a token $x_t$ is considered a potential boundary if its surprise value exceeds a threshold $T$:

依循认知建模工作，我们采用阈值机制对事件边界进行初步识别，这是该机制首次用于 LLM。 形式上，如果 token $x_t$ 的惊奇度超过阈值 $T$，它就被视为一个潜在边界：

$$\begin{array}{rl}-\log P(x_t\mid x_1,\dots ,x_{t-1};\theta )& >T,\\ T& ={\mu }_{t-\tau :t}+\gamma {\sigma }_{t-\tau :t}.\end{array}$$

Here ${\mu }_{t-\tau :t}$ and ${\sigma }_{t-\tau :t}^2$ are the mean and variance of surprise for a window offset $\tau$, and $\gamma$ is a scaling factor. The choice of threshold $T$ is critical in balancing the granularity of segmentation with the model's sensitivity to contextual shifts. If the $T$ is too high, we will identify very few event boundaries, especially if the local context contains few surprising tokens. Conversely, a low $T$ results in frequent boundary identification. Using a moving window ensures that $T$ adapts to contextual shifts, minimizing the need for manual tuning while maintaining control over threshold sensitivity via $\gamma$. This initial segmentation results in a set of potential event boundaries $\mathcal{B}=\{b_1,b_2,\dots ,b_k\}$, where each $b_i$ represents the index of a token exceeding the surprise threshold. These boundaries serve as the starting point for our subsequent refinement process, which aims to optimise the intra-event coherence and inter-event distinctiveness of the resulting memory segments.

其中，${\mu }_{t-\tau :t}$ 和 ${\sigma }_{t-\tau :t}^2$ 是偏移量为 $\tau$ 的窗口中惊奇度的均值和方差，$\gamma$ 是缩放因子。 阈值 $T$ 的选择对于平衡分割粒度与模型对上下文变化的敏感度至关重要。 如果 $T$ 过高，我们识别出的事件边界会非常少，尤其是在局部上下文只包含少量高惊奇度 token 时。 相反，较低的 $T$ 会导致频繁识别边界。 使用移动窗口可以确保 $T$ 适应上下文变化，在通过 $\gamma$ 控制阈值敏感度的同时，最大程度减少人工调节需求。 这一初始分割得到一组潜在事件边界 $\mathcal{B}=\{b_1,b_2,\dots ,b_k\}$，其中每个 $b_i$ 表示超过惊奇度阈值的 token 索引。 这些边界是后续细化过程的起点，该过程旨在优化所得记忆分段的事件内连贯性和事件间区分度。

3.3 Boundary refinement

Algorithm 1: Event segmentation in KV cache

1. Input: tok: List of tokens in the sequence
2. Input: $T$: Threshold for surprisal to identify initial boundaries
3. Input: $f$: Metric function to evaluate potential boundaries
4. Output: $\mathcal{B}$: List of final boundary positions
5. $\mathcal{B}\leftarrow [i\text{for}i\text{in range}(\mathrm{length}(\mathtt{\text{tok}}))$
       $\text{if}-\log P(\mathtt{\text{tok}}[i])>T]$ ▷ Boundary identification
6. for $i$ in $\mathrm{range}(\mathrm{length}(\mathcal{B}))$ do
7. $\alpha ,\beta =\mathcal{B}[i],\mathcal{B}[i+1]$
8. $\mathcal{B}[i+1]\leftarrow \arg \underset{\hat{\beta }\in (\alpha ,\beta ]}{max}f(A,\{\alpha ,\hat{\beta }\})$ ▷ Boundary refinement
9. end for
10. return $\mathcal{B}$

While surprise-based segmentation provides an effective initial estimate of event boundaries, we make the key observation that the utility of elements within an event, during memory recall, depends on their likelihood of being utilised by the current query. Therefore, we theorise that memory recall will be most efficient with high intra-event similarity between keys while maintaining low inter-event similarity. For instance, see the similarity of groups in Figure 2. To further ensure this, we introduce a boundary refinement step that looks to optimise this objective. Such an objective is typically optimised in the context of graph-clustering, hence we express this refinement process in a graph-theoretic manner. To achieve this, we treat the similarity matrix between all keys of an attention head $h$ within the local context window for tokens $x_1,x_2,\dots ,x_n$ as an adjacency matrix $A^h$:

尽管基于惊奇度的分割能够有效初步估计事件边界，我们发现，在记忆回忆过程中，事件内元素的效用取决于它们被当前查询使用的可能性。 因此，我们推断，当键之间的事件内相似度较高，同时保持较低的事件间相似度时，记忆回忆效率最高。 例如，可参见图2中各分组的相似性。 为进一步确保这一点，我们引入一个旨在优化该目标的边界细化步骤。 这种目标通常在图聚类语境下进行优化，因此我们以图论方式表达这一细化过程。 为此，我们把局部上下文窗口内注意力头 $h$ 的所有键之间的相似度矩阵视为 token $x_1,x_2,\dots ,x_n$ 的邻接矩阵 $A^h$：

$$A_{ij}^h=\mathrm{sim}(K_i^h,K_j^h).$$

Here $K_i^h$ and $K_j^h$ are the key vectors corresponding to tokens $x_i$ and $x_j$, respectively. The similarity function measures the closeness of two key vectors; in our implementation, we use dot product similarity ${K_i^h}^{\mathsf{T}}\cdot K_j^h$ due to its effectiveness in capturing semantic relationships in high-dimensional spaces and to align with the mechanism of self-attention in Transformers.

其中，$K_i^h$ 和 $K_j^h$ 分别是与 token $x_i$ 和 $x_j$ 对应的键向量。 相似度函数衡量两个键向量的接近程度；在实现中，我们使用点积相似度 ${K_i^h}^{\mathsf{T}}\cdot K_j^h$，因为它能够有效捕捉高维空间中的语义关系，并与 Transformer 的自注意力机制保持一致。

To evaluate the quality of potential boundaries, we define a metric function $f(A,\mathcal{B}):{\mathbb{R}}^{n\times n}\times \{1,\dots ,n{\}}^k\to \mathbb{R}$. This function quantifies the cohesion within events and separation between events based on the graph structure represented by the similarity matrix $A$ and event boundaries $\mathcal{B}$. We experiment with two widely-accepted graph-clustering metrics: modularity and conductance. Modularity provides a measure of the quality of a particular division of a network into communities, with higher values indicating higher edge density in the identified cluster when compared to the density of edges expected in a random cluster. As our edge weights represent the similarity between two tokens, we seek to maximise this metric. Modularity is defined as:

为评估潜在边界的质量，我们定义指标函数 $f(A,\mathcal{B}):{\mathbb{R}}^{n\times n}\times \{1,\dots ,n{\}}^k\to \mathbb{R}$。 该函数根据相似度矩阵 $A$ 和事件边界 $\mathcal{B}$ 所表示的图结构，量化事件内聚合度和事件间分离度。 我们实验了两种广泛接受的图聚类指标：模块度和电导率。 模块度衡量网络特定社区划分的质量；与随机聚类中预期的边密度相比，数值越高表示识别出的聚类内边密度越高。 由于边权重表示两个 token 之间的相似度，因此我们力求最大化这一指标。 模块度定义为：

$$\begin{array}{rl}{f}_M(A^h,\mathcal{B})& =\frac{1}{4m}\sum _{i,j}[A_{ij}^h-\frac{1}{2m}\left(\sum _i{A}_{ij}^h\right)\left(\sum _j{A}_{ij}^h\right)]\\ & \cdot \delta (c_i,c_j).\end{array}$$

Here $m$ is the total edge weight in the graph, $c_i$ is the community (episodic event) to which node $i$ is assigned, and $\delta$ is the Kronecker delta function. Conductance, on the other hand, measures the fraction of total weighted edges cut by a given community boundary, and is defined as:

其中，$m$ 是图中的总边权重，$c_i$ 是节点 $i$ 被分配到的社区（情景事件），$\delta$ 是克罗内克 delta 函数。 另一方面，电导率衡量给定社区边界切断的加权边占总加权边的比例，其定义为：

$$\begin{array}{rl}{f}_C(A^h,\mathcal{B})& =\underset{S\in V}{min}\frac{\sum _{i\in S,j\notin S}{A}_{ij}^h}{min(\mathrm{vo}(S),\mathrm{vo}(V\setminus S))},\\ \mathrm{vo}(S)& =\sum _{i,j\in S}{A}_{ij},\\ \mathrm{vo}(V\setminus S)& =\sum _{i,j\notin S}{A}_{ij}.\end{array}$$

Here $S=\{b_i,b_i+1,\dots ,b_{i+1}\}$ is a subset of all nodes $V=\{b_1,b_1+1,\dots ,b_k\}$ in the induced graph, with $b_i\in \mathcal{B}$. Lower conductance values indicate better community structure. Our boundary refinement algorithm sequentially adjusts the initial surprise-based boundaries to optimise these metric functions. While our best results are achieved using modularity, we also include comparisons with conductance-based boundary refinement to provide a comprehensive analysis. The overall process is summarized in Algorithm 1 and further discussed in the appendix.

其中，$S=\{b_i,b_i+1,\dots ,b_{i+1}\}$ 是诱导图全部节点 $V=\{b_1,b_1+1,\dots ,b_k\}$ 的子集，且 $b_i\in \mathcal{B}$。 电导率越低，表示社区结构越好。 我们的边界细化算法依次调整基于惊奇度得到的初始边界，以优化这些指标函数。 尽管使用模块度取得了最佳结果，我们也加入了基于电导率的边界细化比较，以提供全面分析。 整体过程总结于算法1，附录中有进一步讨论。

This algorithm first identifies initial boundaries based on the surprise threshold $T$, then refines these boundaries by finding the optimal position $\hat{\beta }$ between each pair of consecutive initial boundaries $(\alpha ,\beta )$ that optimises the chosen metric function $f$ (either maximising modularity or minimising conductance). This process ensures that the final segmentation (1) captures points of high surprise and (2) optimises for coherent information grouping. The boundary identification step incurs negligible computational cost, as it only evaluates existing LLM outputs. The time complexity of Algorithm 1 has an overall complexity of $\mathcal{O}(nm)$, where $n$ is the $n$ is the sequence length and $m$ is the chunk size selected to process the sequence (for details see the appendix).

该算法首先根据惊奇度阈值 $T$ 识别初始边界，再在每对连续初始边界 $(\alpha ,\beta )$ 之间找到使所选指标函数 $f$ 最优的位置 $\hat{\beta }$，从而细化这些边界；所选目标可以是最大化模块度或最小化电导率。 这一过程确保最终分割（1）捕捉高惊奇度位置，并（2）优化信息的连贯分组。 边界识别步骤只评估现有 LLM 输出，因此计算成本可以忽略。 算法1的整体时间复杂度为 $\mathcal{O}(nm)$，其中 $n$ 是序列长度，$m$ 是处理序列时选择的分块大小；细节见附录。

3.4 Memory Retrieval

When inferring a new token, a number of episodic events are selected and become a part of the (extended) context window of the underlying LLM. Our memory retrieval process employs a two-stage mechanism to select relevant episodic events for the LLM's context window (Figure 3C). First, we retrieve $k_s$ events using k-NN search based on dot product similarity between the current query and representative tokens of each event. These representatives, selected as per Xiao et al., are the most influential tokens within each event. For large memory stores, we utilise approximate k-NN to maintain efficiency. These $k_s$ events, retrieved based on their similarity to the current query, form a part of the LLM's context window that we refer to as the similarity buffer.

推断新 token 时，系统会选择若干情景事件，并把它们纳入基础 LLM 的扩展上下文窗口。 我们的记忆检索过程采用两阶段机制，为 LLM 的上下文窗口选择相关情景事件（图3C）。 首先，我们根据当前查询与每个事件的代表 token 之间的点积相似度，使用 k-NN 搜索检索 $k_s$ 个事件。 按照 Xiao 等人的方法，这些代表 token 是各事件中最具影响力的 token。 对于大型记忆存储，我们使用近似 k-NN 来保持效率。 这 $k_s$ 个根据与当前查询的相似度检索出的事件构成 LLM 上下文窗口的一部分，我们称之为相似度缓冲区。

The second stage of our retrieval process introduces another buffer, which we refer to as the contiguity buffer, designed to maintain temporal context. Implemented as a queue of size $k_c$, this buffer promotes temporal relationships in retrieval. When an event is retrieved, we also enqueue its neighboring events (within $\pm n$ positions in the original sequence) into this buffer. This mechanism enables the LLM's ``induction'' attention heads to exhibit the contiguity and asymmetry effects discussed in Section 2.2. The queue structure allows for a natural decay of temporal context as new events are processed, with older or repeated events being dequeued as new ones are added. In total, $k=k_s+k_c$ events are added to the context window, striking a balance between relevance and temporal relationships in a manner analogous to human episodic memory retrieval. Note that each layer retrieves and attends to these $k$ events individually, allowing it to potentially focus on different parts of the context.

检索过程的第二阶段引入另一个旨在维持时间上下文的缓冲区，我们称之为连续性缓冲区。 该缓冲区实现为大小为 $k_c$ 的队列，用于促进检索中的时间关系。 检索到一个事件时，我们还会把它在原始序列中 $\pm n$ 个位置范围内的相邻事件加入该缓冲区。 这一机制使 LLM 的“归纳”注意力头能够表现出第 2.2 节讨论的连续性和不对称性效应。 随着新事件被处理，队列结构会使时间上下文自然衰减；添加新事件时，较旧或重复的事件会出队。 总共有 $k=k_s+k_c$ 个事件被加入上下文窗口，以类似人类情景记忆检索的方式平衡相关性与时间关系。 需要注意的是，每层都会单独检索并关注这 $k$ 个事件，因此可能聚焦于上下文的不同部分。

4. Experiments

4.1 Performance of EM-LLM on long-context tasks

Comparison with KV-retrieval-based LLMs At the time of writing, InfLLM is considered to be the SOTA KV-retrieval method on long-context benchmarks (LongBench, $\mathrm{\infty }$-Bench), as well as being the only method which uses group-based k-NN retrieval in LLMs on such benchmarks. We, therefore, employ this model as our first baseline for comparison with our own methods.

与基于 KV 检索的 LLM 比较。 在本文撰写时，InfLLM 被认为是长上下文基准（LongBench、$\mathrm{\infty }$-Bench）上最先进的 KV 检索方法，也是唯一在这些基准上使用 LLM 分组 k-NN 检索的方法。 因此，我们把该模型作为与自身方法比较的第一个基线。

表1：EM-LLM 在 LongBench（分组任务）和 ∞-Bench 上与基线 InfLLM 的性能比较。S：惊奇度阈值；SM：惊奇度阈值与模块度细化；S+C：惊奇度阈值与连续性缓冲区；SM+C：惊奇度、细化与连续性缓冲区。每行括号中给出 InfLLM 和 EM-LLM 使用的局部 token 数与检索 token 数。

Base LLM	Method	LongBench						Avg.	∞-Bench
		SQA	MQA	Sum	FSL	Ret	Cod		C.D	M.F	MC	R.KV	R.P	R.N
Mistral v2	InfLLM (4k+2k)	33	25.5	27.1	66.1	64	54.8	41.9	29.4	26.6	43.2	95.6	100	99.8
	EM-LLMSM+C	32.9	27	27.2	66.8	84.1	54.8	43.7	28.2	27.1	42.8	99	100	99.8
LLaMA 3	InfLLM (4k+4k)	38.5	36.9	27	69	84	53.2	47	30.5	23.7	43.7	5	100	99
	EM-LLMS	39.3	37.7	27.0	69.2	87.5	50.3	47.2	31.7	16.9	40.6	4.2	100	99.5
LLaMA 3.1	InfLLM (4k+4k)	41.4	40.7	29	69	97	64.2	51.1	22.6	33.7	46.7	81	100	100
	EM-LLMSM	41.2	41.3	29.2	69.1	98.5	64.1	51.3	22.6	34	47.6	90.2	100	100
Phi 3	InfLLM (1k+3k)	28.4	24.9	25.6	52.9	7.5	57	34.5
	EM-LLMS	29.2	27.1	25.9	53.5	10	57	35.4
Phi 3.5	InfLLM (1k+3k)	31.7	28.5	23.9	56.3	11.5	40.3	34.2
	EM-LLMS	31.8	31.9	24.5	55.5	13	39.5	34.9

Results on both benchmarks (Table 1) show that our method is able to improve on InfLLM across 5 different base LLMs, $80\mathrm{\%}$ of individual task groups of LongBench and on the overall average. Note that the table shows the best single method in terms of overall performance for each ablation (see the appendix for all ablations in methods). Looking at individual task performance across all ablations in methods, EM-LLM is able to surpass InfLLM in all tasks. Notably, we see an especially large jump in performance in the retrieval (Passage, KV, Passkey, Number) and QA (Narrative, Qasper, MultiField, Hotpot, 2Wiki and Musique) tasks across all ablations, with up to a $40\mathrm{\%}$ and $29.7\mathrm{\%}$ improvement over InfLLM respectively. Such tasks require the model to identify and retrieve specific information within the input sequence, a challenging test for the model's ability to accurately recall a wide range of detailed information from a large context concurrently. This substantial improvement highlights the effectiveness of our event segmentation method in enhancing long-term memory recall and retrieval accuracy in LLMs.

两个基准上的结果（表1）表明，我们的方法在 5 种不同基础 LLM、LongBench 的 $80\mathrm{\%}$ 独立任务组以及总体平均结果上均优于 InfLLM。 需要注意的是，该表展示了每种消融设置中总体性能最好的单一方法；所有方法消融见附录。 考察所有方法消融下的各任务性能，EM-LLM 能够在所有任务上超过 InfLLM。 尤其值得注意的是，在所有消融设置下，检索任务（Passage、KV、Passkey、Number）和问答任务（Narrative、Qasper、MultiField、Hotpot、2Wiki 和 Musique）的性能跃升特别明显，相比 InfLLM 最高分别提升 $40\mathrm{\%}$ 和 $29.7\mathrm{\%}$。 这类任务要求模型识别并检索输入序列中的特定信息，是对模型能否同时从大型上下文中准确回忆广泛细节信息的一项高难度测试。 这一显著提升凸显了事件分割方法在增强 LLM 长期记忆回忆和检索准确率方面的有效性。

Comparison with RAG and full-context LLMs To evaluate EM-LLM against prominent methods for handling long contexts, we compared its performance on LLaMA 3.1-8B with two different RAG approaches, including the current SOTA NV-Embed-v2 retriever, as well as with the brute-force baseline of processing all tokens directly within the LLM's softmax attention (full-context). Across most tasks in our benchmarks, EM-LLM outperformed both RAG and full-context methods, as well as a custom surprise-based RAG method (Figure 1 and the appendix), exceeding the performance of NV-Embed-v2 by $30.5\mathrm{\%}$ on LongBench and by $11.5\mathrm{\%}$ on $\mathrm{\infty }$-Bench.

与 RAG 和完整上下文 LLM 比较。 为了针对主流长上下文处理方法评估 EM-LLM，我们在 LLaMA 3.1-8B 上把它与两种不同的 RAG 方法进行比较，其中包括当前最先进的 NV-Embed-v2 检索器；我们还与一种暴力基线进行比较，该基线直接在 LLM 的 softmax 注意力中处理所有 token，即完整上下文。 在基准的大多数任务上，EM-LLM 均优于 RAG 和完整上下文方法，也优于一种自定义的基于惊奇度的 RAG 方法（见图1和附录）；在 LongBench 和 $\mathrm{\infty }$-Bench 上，它分别比 NV-Embed-v2 高 $30.5\mathrm{\%}$ 和 $11.5\mathrm{\%}$。

This significant performance boost over RAG can be attributed to EM-LLM's ability to retrieve and incorporate relevant information at each layer individually, rather than relying on a single retrieval step as in RAG (for an illustration, see the supplementary figure). By accessing more specific and contextually relevant information through layer-wise key-value retrieval, EM-LLM effectively addresses RAG's limitations in precision and lower overall performance. Additionally, EM-LLM's hierarchical attention avoids the issue of diluted attention in large context windows that affects full-context models, enabling it to outperform both RAG and full-context LLMs on the LongBench dataset. Furthermore, EM-LLM demonstrated remarkable scalability by achieving $100\mathrm{\%}$ accuracy on the Passkey.Retrieval task with sequences up to $10.2$M tokens, far beyond the practical limits of full-context LLMs. This highlights EM-LLM's efficiency in handling extremely long contexts, positioning it as a powerful alternative for long-context processing.

EM-LLM 相比 RAG 的显著性能提升可以归因于它能够在每一层分别检索并引入相关信息，而不是像 RAG 那样依赖单次检索步骤；补充图中给出了示意。 通过逐层键值检索访问更具体且与上下文更相关的信息，EM-LLM 有效解决了 RAG 在精度和总体性能较低方面的局限。 此外，EM-LLM 的层次注意力避免了影响完整上下文模型的大型上下文窗口注意力稀释问题，使其能够在 LongBench 数据集上同时超过 RAG 和完整上下文 LLM。 此外，EM-LLM 在长度高达 $10.2$M token 的序列上完成 Passkey.Retrieval 任务并达到 $100\mathrm{\%}$ 准确率，展现了远超完整上下文 LLM 实际极限的显著可扩展性。 这凸显了 EM-LLM 处理极长上下文的效率，使其成为长上下文处理的有力替代方案。

4.2 Human and LLM surprise cluster similar tokens together

As mentioned in Section 3.2, we employ modularity and conductance as two refinement objectives in our boundary refinement algorithm, due to their qualities in assessing the intra- and inter-event similarities between individual tokens. We will now use such metrics to compare various event segmentation methods, including human event segmentation data. Additionally, we introduce one further, simple metric for this experiment: the ratio between intra- and inter-community similarity (I/IS), calculated for each head and community $S$ as follows:

如第 3.2 节所述，由于模块度和电导率能够评估各 token 之间的事件内与事件间相似度，我们在边界细化算法中把它们用作两个细化目标。 现在，我们将使用这些指标比较包括人类事件分割数据在内的多种事件分割方法。 此外，我们为该实验引入另一个简单指标：社区内与社区间相似度之比（I/IS），针对每个注意力头和社区 $S$ 计算如下：

$$\begin{array}{rl}\mathrm{intra}& =\sum _{i\in S,j\in S}{A}_{ij},\\ \mathrm{inter}& =\sum _{i\in S,j\notin S}{A}_{ij},\\ \mathrm{I}/\mathrm{IS}& \equiv \frac{\mathrm{intra}}{\mathrm{inter}}.\end{array}$$

Kumar et al. found strong correlations between human-perceived events and prediction errors across 3 short podcasts (7-30 minutes), when processing the corresponding transcript with an LLM. Taking advantage of such human-annotated data and results from previous studies on this dataset, we compare the segmentation quality and correlation with human segmentation for each of our methods (Figure 4) using our similarity metrics.

Kumar 等人在使用 LLM 处理相应文字稿时，发现 3 个短播客（7 至 30 分钟）中的人类感知事件与预测误差之间存在强相关性。 利用这些人工标注数据以及此前研究在该数据集上的结果，我们使用相似度指标比较每种方法的分割质量及其与人类分割的相关性（图4）。

图4：在人工标注音频数据集中比较人类事件分割与不同计算分割方法。（A）各 LLaMA2 层 KV cache 聚合度和分离度指标的差异，报告每种方法与对应随机分割的差值。（B）人类报告与不同方法之间的距离。固定方法 F、FM、FC 的表现低于对应的惊奇度方法 S、SM、SC，且 InfLLM 的 F 方法低于随机分割。

As shown in Figure 4A, human-perceived events achieve significantly higher scores in similarity metrics compared to fixed or random events, suggesting that surprise is indeed an important factor for humans in their own perception of events. Furthermore, surprise-only segmentation ($\mathbf{S}$) achieves very similar results to humans, while the addition of our refinement algorithm ($\mathbf{SM}$, $\mathbf{SC}$, $\mathbf{FM}$, $\mathbf{FC}$) significantly improves performance. Figure 4B further shows that surprise-based methods ($\mathbf{S}$, $\mathbf{SM}$, $\mathbf{SC}$), consistently identify event boundaries that are closest to those perceived by humans.

如图4A 所示，与固定或随机事件相比，人类感知事件在相似度指标上取得了明显更高的分数，这表明惊奇度确实是人类感知事件时的一个重要因素。 此外，仅使用惊奇度的分割（$\mathbf{S}$）取得了与人类非常相似的结果，而加入我们的细化算法（$\mathbf{SM}$、$\mathbf{SC}$、$\mathbf{FM}$、$\mathbf{FC}$）则显著提升了性能。 图4B 进一步表明，基于惊奇度的方法（$\mathbf{S}$、$\mathbf{SM}$、$\mathbf{SC}$）始终能识别出最接近人类感知结果的事件边界。

4.3 Comparing segmentation methods

Our experiments on the PG-19 dataset (see Table 2) clearly demonstrate that surprise-based segmentation with refinement ($\mathbf{SM}$, $\mathbf{SC}$) provides the best results in terms of event similarity metrics, regardless of the base LLM used. While the surprise-only method ($\mathbf{S}$) achieves decent results, we observe that refinement is especially adept to improving this performance with regards to our metrics, as it is directly optimising for such an objective. Interestingly however, the fixed-based refinement methods ($\mathbf{FM}$, $\mathbf{FC}$) do not reach the same performance as their surprise-based counterparts, further showing that the initial segmentation with a surprise threshold is crucial to achieving the best possible balance in intra-/inter-similarity with our methods.

我们在 PG-19 数据集上的实验（见表2）清楚表明，无论使用哪种基础 LLM，带细化的惊奇度分割（$\mathbf{SM}$、$\mathbf{SC}$）都在事件相似度指标上取得了最佳结果。 尽管仅使用惊奇度的方法（$\mathbf{S}$）取得了不错的结果，我们观察到细化特别善于提升这些指标，因为它直接优化这一目标。 然而有趣的是，基于固定分割的细化方法（$\mathbf{FM}$、$\mathbf{FC}$）没有达到基于惊奇度的对应方法的性能，这进一步表明，使用惊奇度阈值进行初始分割对于让我们的方法在事件内与事件间相似度之间取得尽可能好的平衡至关重要。

4.4 Similarity, Contiguity, Recency and Temporal Order

表2：在 PG-19 数据集和 γ = 10⁻³ 设置下，比较不同 LLM 与分割方法在 KV cache 上的图论指标。报告值为相对随机分割的差值。Mod：模块度 × 10⁵；Con：电导率；I/IS：事件内/事件间相似度 × 10³。

LLM	Metric	F	FM	FC	S	SM	SC
Mistral-7B	Mod ↑	-2.3 ± 4.1	29.2 ± 44.0	6.7 ± 25.9	18.6 ± 29.6	39.9 ± 55.5	29.5 ± 42.7
	Con ↓	9.1 ± 8.7	-16.9 ± 6.7	-12.5 ± 9.6	-23.6 ± 9.4	-24.6 ± 9.3	-27.6 ± 9.8
	I/IS ↑	-4.3 ± 4.0	31.2 ± 21.4	3.7 ± 14.9	17.9 ± 17.0	35.3 ± 27.7	21.6 ± 22.4
LLaMA2-7B	Mod ↑	-1.1 ± 4.3	13.4 ± 19.5	0.6 ± 7.3	8.7 ± 16.0	18.7 ± 26.4	11.5 ± 19.4
	Con ↓	11.9 ± 9.8	-18.8 ± 7.4	-13.7 ± 10.9	-29.5 ± 10.2	-29.7 ± 10.1	-33.3 ± 10.3
	I/IS ↑	-3.8 ± 3.7	20.7 ± 184.7	-1.1 ± 6.8	15.0 ± 880.0	25.0 ± 19.9	16.5 ± 15.4
LLaMA3-8B	Mod ↑	-1.6 ± 3.6	18.9 ± 25.6	0.9 ± 11.8	13.1 ± 21.5	27.0 ± 35.6	18.3 ± 28.5
	Con ↓	11.3 ± 9.5	-20.3 ± 6.9	-14.6 ± 11.4	-29.7 ± 9.2	-30.6 ± 9.2	-33.9 ± 9.6
	I/IS ↑	-3.8 ± 3.1	24.5 ± 13.9	-1.1 ± 5.8	15.7 ± 11.0	28.1 ± 16.1	16.4 ± 12.2

As demonstrated in Table 1 and Table 2, along with Figure 4, each of our ablations show various positive improvements on InfLLM. As mentioned in Section 4.3, refinement has a strong positive impact in improving our similarity metrics. This is seen to translate well to model performance in our experiments, with the addition of refinement achieving the best performance in $60\mathrm{\%}$ of tasks across LongBench and $\mathrm{\infty }$-Bench, as well as agreeing with human data (Figure 4). The effects of contiguity are also clearly demonstrated, with the addition of our contiguity buffer achieving the best performance in $44\mathrm{\%}$ of tasks. Furthermore, these methods are seen to be complementary, often improving on both individual additions.

如表1、表2和图4所示，我们的各项消融都对 InfLLM 带来了不同程度的正向改进。 如第 4.3 节所述，细化对提升相似度指标具有强烈的正向作用。 这在实验中良好转化为模型性能：加入细化后，在 LongBench 和 $\mathrm{\infty }$-Bench 的 $60\mathrm{\%}$ 任务上取得了最佳性能，并且与人类数据一致（图4）。 连续性的作用也得到了清楚展示：加入连续性缓冲区后，在 $44\mathrm{\%}$ 的任务上取得了最佳性能。 此外，这些方法表现出互补性，结合后通常会优于任一单独改进。

However, the fact that certain tasks still appear to benefit more from either surprise-only, refinement, or contiguity, is an interesting result. This is likely due to the nature of the tasks and the varying importance of contiguity across these tasks. Where contiguity is not crucial, adding such a buffer to our context window also reduces the size of the similarity buffer, and therefore provides potentially less directly relevant events. This is compatible with our own findings that a contiguity buffer that is as big or smaller than the similarity buffer yields the best results, suggesting that the similarity buffer is still the most crucial part of our approach. This is especially the case when combined with refinement, which we expect is due to the improved similarity of refined events, hence further reducing the need for contiguous events.

然而，一些任务似乎仍然更受益于仅使用惊奇度、细化或连续性中的某一种，这是一个有趣结果。 这可能源于任务本身的性质，以及连续性在不同任务中的重要程度不同。 在连续性并不关键的任务中，向上下文窗口添加这种缓冲区还会缩小相似度缓冲区，因此可能提供更少的直接相关事件。 这与我们的发现一致：与相似度缓冲区一样大或更小的连续性缓冲区能取得最佳结果，这表明相似度缓冲区仍是我们方法中最关键的部分。 与细化结合时尤其如此；我们认为，这是因为细化后事件的相似度得到提高，从而进一步降低了对连续事件的需求。

5. Discussion

Human studies Significant correlations have been found between human event segmentation and prediction errors in both LLMs and video models. Our results add to this growing body of evidence, demonstrating that LLM-based surprise can serve as a proxy for human event segmentation, in multiple levels of hierarchical abstraction, and that the resulting event structure in EM-LLM's attention heads correlates strongly with human-perceived events. This finding suggests a potential, low-level parallels between LLM mechanisms and human cognitive processes.

人类研究。 在 LLM 和视频模型中，人类事件分割与预测误差之间都已被发现存在显著相关性。 我们的结果进一步扩充了这些不断积累的证据，表明基于 LLM 的惊奇度能够在多层次抽象中作为人类事件分割的替代指标，并且 EM-LLM 注意力头中由此形成的事件结构与人类感知事件高度相关。 这一发现表明，LLM 机制与人类认知过程之间可能存在低层次的相似性。

Furthermore, our model's use of both similarity-based and temporally contiguous retrieval mechanisms parallels human memory retrieval patterns, allowing for the expression of robust phenomena found in human memory research. The temporal contiguity effect, where items experienced close together in time are often recalled together, is a robust phenomenon in human memory research. Further experiments could deepen our understanding of the connections between EM-LLM and human episodic memory. Following Michelmann et al., one could test whether the timing of the event boundaries or the degree of modularity per level that our method produces is closer on average to the human consensus, than individual human subjects. Additionally, exploring how different ratios of the contiguity buffer affect the reproduction of human memory biases, and investigating the impact of recency and initial surprise on event recall, could reveal the extent to which EM-LLM exhibits biases found in free recall studies.

此外，我们的模型同时采用基于相似度和时间连续性的检索机制，这与人类记忆检索模式相似，使其能够表现出人类记忆研究中发现的稳健现象。 时间连续性效应是人类记忆研究中的稳健现象，即在时间上相近经历的项目通常会被一同回忆。 进一步实验可以加深我们对 EM-LLM 与人类情景记忆之间联系的理解。 依循 Michelmann 等人的工作，可以检验我们的方法所产生的事件边界时机或每层模块度，平均而言是否比单个人类受试者更接近人类共识。 此外，探索连续性缓冲区的不同比例如何影响人类记忆偏差的复现，并研究近因性和初始惊奇度对事件回忆的影响，可以揭示 EM-LLM 在多大程度上表现出自由回忆研究中发现的偏差。

Furthermore, EM-LLM's architecture with differentiated context handling (Section 3.1) invites comparisons to cognitive models of human memory beyond episodic. The local context, holding recent and task-relevant information, resembles the limited-capacity working memory system described by Baddeley. Given that EM-LLM's broader context window includes both local context and retrieved memories, it aligns more closely with Ericsson and Kintsch's concept of long-term working memory, which allows rapid access to relevant long-term information beyond traditional capacity limits. Alternatively, our architecture parallels Cowan's embedded-processes model, where the local context is the ``focus of attention'', and the full context window represents the activated portion of long-term memory. Future work could explore these analogies further, using EM-LLM as a test-bed for hypotheses about human memory and working memory capacity limits. Inspired by Baddeley's multi-component model, integrating modality-specific buffers into EM-LLM might enhance performance on multi-modal tasks.

此外，EM-LLM 采用差异化上下文处理的架构（第 3.1 节），也适合与情景记忆以外的人类记忆认知模型进行比较。 保存近期任务相关信息的局部上下文类似于 Baddeley 描述的容量有限的工作记忆系统。 鉴于 EM-LLM 更广泛的上下文窗口同时包含局部上下文和检索到的记忆，它更接近 Ericsson 和 Kintsch 的长期工作记忆概念，后者允许人们超越传统容量限制，快速访问相关长期信息。 另一方面，我们的架构与 Cowan 的嵌入过程模型相似，其中局部上下文是“注意焦点”，完整上下文窗口则表示长期记忆中被激活的部分。 未来工作可以进一步探索这些类比，把 EM-LLM 用作检验人类记忆和工作记忆容量限制假设的试验平台。 受 Baddeley 多组件模型启发，把特定模态缓冲区集成到 EM-LLM 中可能会提升多模态任务的性能。

Machine learning In refining event boundaries, we utilised modularity and conductance as metrics for evaluating community structure in the similarity graph of attention keys. While effective in our experiments, we acknowledge that numerous other methods for graph clustering and sequence segmentation could potentially be applied. Our choice was motivated by their established theoretical foundations and computational efficiency, though comparative studies suggest performance can vary based on network characteristics. Interestingly, our surprise-based initial boundary detection shares similarities with Bayesian online change-point detection, suggesting potential avenues for integrating time series analysis techniques into LLM context processing. Future work could explore whether more sophisticated segmentation or clustering algorithms could improve EM-LLM's performance, particularly for extremely long contexts or streaming data scenarios. Such investigations could enhance our model and contribute to understanding how information is structured and processed in LLMs, bridging the gap between traditional sequence analysis and LLM context processing.

机器学习。 在细化事件边界时，我们使用模块度和电导率作为评估注意力键相似图中社区结构的指标。 尽管它们在实验中有效，我们也承认还可以采用许多其他图聚类和序列分割方法。 我们选择这些指标是因为它们具有成熟的理论基础和较高计算效率，不过比较研究表明，性能可能随网络特征而变化。 有趣的是，我们基于惊奇度的初始边界检测与贝叶斯在线变点检测存在相似之处，这表明可以把时间序列分析技术集成到 LLM 上下文处理中。 未来工作可以探索更复杂的分割或聚类算法能否提升 EM-LLM 的性能，尤其是在极长上下文或流式数据场景中。 这类研究可以增强我们的模型，帮助理解信息在 LLM 中如何组织和处理，并弥合传统序列分析与 LLM 上下文处理之间的差距。

Looking ahead, promising directions for future research include extending our segmentation processes to operate at each layer of the Transformer independently. This could lead to more nuanced and hierarchical representations of episodic memories, following the underlying semantic structure of the input more closely. Additionally, exploring how EM-LLM could be utilised to enable imagination and future thinking has great potential for advancing model-based reinforcement learning and continual learning techniques in LLMs. By leveraging its event-based structure to simulate potential future scenarios or recall past experiences in novel contexts, EM-LLM could enhance an LLM's ability to plan, adapt, and learn continuously from new information.

展望未来，一个有前景的研究方向是扩展分割过程，使其能够在 Transformer 的每一层独立运行。 这可能产生更细致且更具层次性的情景记忆表示，更紧密地遵循输入的底层语义结构。 此外，探索如何利用 EM-LLM 实现想象和未来思考，对于推进 LLM 中基于模型的强化学习和持续学习技术具有巨大潜力。 通过利用基于事件的结构模拟潜在未来情景，或在新情境中回忆过去经历，EM-LLM 可以增强 LLM 规划、适应并持续从新信息中学习的能力。

6. Conclusion

In this work, we introduced EM-LLM, a flexible architecture that integrates key aspects of human episodic memory and event cognition into Transformer-based LLMs. Our approach enables existing LLMs to effectively process vastly extended contexts without the need for pre-training, demonstrating superior performance on long-context tasks compared to the corresponding SOTA. By combining surprise-based event segmentation, graph-theoretic boundary refinement, and a two-stage memory retrieval process, EM-LLM offers a promising path toward virtually infinite context windows. This capability has the potential to revolutionize interactions with LLMs, enabling continuous, personalised exchanges over extended periods and serving as a viable alternative to traditional RAG techniques. Finally, by bridging insights from cognitive science with machine learning, our approach not only enhances the performance of LLMs on long-context tasks but also provides a scalable framework for computational modelling of episodic and event cognition. We hope this study inspires the community to expand research at the intersection of LLMs and human memory.

在这项工作中，我们提出 EM-LLM，这是一种把人类情景记忆和事件认知的关键方面融入 Transformer LLM 的灵活架构。 我们的方法使现有 LLM 无需预训练即可有效处理大幅扩展的上下文，并在长上下文任务上表现出优于相应最先进方法的性能。 通过结合基于惊奇度的事件分割、图论边界细化和两阶段记忆检索过程，EM-LLM 为实现几乎无限的上下文窗口提供了一条有前景的路径。 这一能力有望彻底改变与 LLM 的交互，实现长时间持续的个性化交流，并成为传统 RAG 技术的可行替代方案。 最后，通过连接认知科学与机器学习的见解，我们的方法不仅增强了 LLM 在长上下文任务上的性能，还为情景认知和事件认知的计算建模提供了一个可扩展框架。 我们希望这项研究能够激励社区拓展 LLM 与人类记忆交叉领域的研究。