---

EM²Mem: Event-Centric Multimodal Memory for Large Language Models

MemoryMultimodalVideo QAEMNLP 2026 Findings浙江大学华南理工大学联想

Chen Y, Zheng Y, Li Y, et al. EM²Mem: event-centric multimodal memory for large language models[PP/OL]. V1. arXiv (2026-09-01)[2026-10-29]. DOI:10.48550/arXiv.2609.00551.

https://arxiv.org/abs/2609.00551v1

https://github.com/zjunlp/LightMem

---

EM²Mem：面向大语言模型的事件中心多模态记忆

Abstract

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM²Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM²Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by $4.67\times$ and total inference tokens by $63.66\mathrm{\%}$.

多模态记忆为长视频问答提供了一种可扩展的接口，但现有方法往往将视频描述、帧、转录文本、摘要或图事实作为彼此孤立的片段进行检索。 这些片段虽然可以被检索，却无法直接用于生成：语言模型必须在推理阶段重建跨模态和时间上的对齐关系，而此时上下文有限，证据归因也很困难。 我们提出 EM²Mem，一种以事件为中心的多模态记忆框架，在记忆构建阶段便将异构证据绑定到事件锚点。 每个由事件索引的记忆单元都对齐了多模态记录、时间上下文、图关联关系、语义事实和来源信息，从而能够围绕有证据支撑的多模态事件进行紧凑的证据读取，而不是读取特定模态的孤立片段。 在三个长视频问答基准上，EM²Mem 相较最强记忆基线的平均准确率分别提高了 2.0、2.4 和 3.7 个百分点，严格事件级 Top-5 证据召回率提高了 7.0 个百分点，单次查询延迟降低至原来的 $1/4.67$，推理 token 总量减少了 $63.66\mathrm{\%}$。

1. Introduction

Multimodal memory is becoming essential for enabling Large Language Models (LLMs) to reason over extended multimodal experiences. Unlike short-image or short-video understanding, long-video QA requires models to answer questions whose supporting evidence may appear sparsely across minutes or hours of visual observations, speech transcripts, OCR, scenes, objects, and recurring behaviors. Since such evidence cannot be reliably captured by a single context window or a flat video representation, LLMs need an external memory that can store observations, retrieve task-relevant context, and support grounded generation. The key problem, is not only how much information to store, but how to organize heterogeneous evidence into retrievable units that align with the way questions are asked and answers are justified.

多模态记忆正成为使大语言模型（LLM）能够对长时间跨度的多模态经历进行推理的关键。 不同于短图像或短视频理解，长视频问答要求模型回答的问题，其支撑证据可能稀疏地分布在数分钟乃至数小时的视觉观测、语音转录、OCR、场景、对象和反复出现的行为之中。 由于单个上下文窗口或扁平的视频表示无法可靠地涵盖这些证据，LLM 需要一种外部记忆，以存储观测、检索与任务相关的上下文，并支持有证据依据的生成。 关键问题不仅在于存储多少信息，还在于如何将异构证据组织为可检索单元，使其符合提问和论证答案的方式。

图1：EM²Mem 将多模态证据组织为以事件为中心的记忆单元，使检索围绕事件展开并具有证据依据，而非检索彼此孤立的片段。

图1：EM²Mem 将多模态证据组织为以事件为中心的记忆单元，使检索围绕事件展开并具有证据依据，而非检索彼此孤立的片段。

Recent multimodal memory and video retrieval-augmented generation methods typically construct fragment-centric memories by compressing videos into modality- or structure-specific stores, such as captions, summaries, keyframes, transcript segments, embeddings, triples, or knowledge graphs. Although these fragments are searchable, they often fail to provide the event-level grounding needed for long-video QA. Questions commonly refer to events, situations, recurring behaviors, and temporally grounded relations, rather than isolated frames, sentences, or graph edges. As a result, existing systems face a retrieve-then-align bottleneck: they first retrieve disconnected fragments and then rely on the LLM to reconstruct missing cross-modal and temporal bindings during generation, when the context budget is limited and attribution is the most difficult to maintain.

近期的多模态记忆和视频检索增强生成方法，通常将视频压缩为特定模态或结构的存储形式，例如视频描述、摘要、关键帧、转录片段、嵌入、三元组或知识图谱，以此构建以片段为中心的记忆。 尽管这些片段可以被检索，但它们往往无法提供长视频问答所需的事件级证据依据。 问题通常涉及事件、情境、反复出现的行为以及具有时间依据的关系，而不是孤立的帧、句子或图边。 因此，现有系统面临 先检索、后对齐 的瓶颈：它们先检索彼此脱节的片段，再依靠 LLM 在生成阶段重建缺失的跨模态绑定和时间绑定，而此时上下文预算有限，证据归因也最难保持。

To address this bottleneck, we rethink the retrieval unit of multimodal memory: How should multimodal memory organize heterogeneous evidence for LLM agents? Inspired by event cognition, which views events as coherent units for organizing perception, language, actions, and temporal boundaries, we propose EM²Mem, an Event-Centric Multimodal Memory framework for long-video QA. Rather than treating events as merely temporal video segments, EM²Mem uses event anchors as language-addressable memory indices that bind captions, transcripts, keyframes, structured metadata, temporal context, and provenance during memory construction. This yields an align-then-retrieve design: heterogeneous evidence is unified into event-indexed multimodal memory cells before retrieval, so the model retrieves grounded event-level evidence rather than assembling isolated fragments afterward. To support reasoning beyond individual events, EM²Mem further builds an episodic graph for cross-event relations and temporal transitions, together with a semantic graph for long-term facts, habits, preferences, and stable relations, each grounded in supporting event evidence.

为解决这一瓶颈，我们重新思考多模态记忆的检索单元：多模态记忆应当如何为 LLM 智能体组织异构证据？ 事件认知将事件视为组织感知、语言、动作和时间边界的连贯单元；受此启发，我们提出 EM²Mem，一个面向长视频问答的 事件中心多模态记忆 框架。 EM²Mem 并非仅将事件视为视频的时间片段，而是使用事件锚点作为可通过语言寻址的记忆索引，在记忆构建阶段将视频描述、转录文本、关键帧、结构化元数据、时间上下文和来源信息绑定在一起。 由此形成了 先对齐、后检索 的设计：在检索之前，就将异构证据统一到由事件索引的多模态记忆单元中，使模型检索的是有依据的事件级证据，而不是事后再拼接孤立的片段。 为支持超越单个事件的推理，EM²Mem 还构建了用于跨事件关系和时间转移的情景图，以及用于长期事实、习惯、偏好和稳定关系的语义图，其中每项内容都以支撑它的事件证据为依据。

This event-centric design yields consistent gains in accuracy, efficiency, and attribution. On three long-video QA benchmarks, EgoLifeQA, Ego-R1 Bench, and Video-MME (L), EM²Mem improves average accuracy over the strongest memory baseline by 2.0%, 2.4%, and 3.7% respectively, while reducing per-query latency by 4.67 times and total inference tokens by 63.66%. It also localizes evidence more precisely, improving strict event-level Top-5 recall by 7.0 points over WorldMM's five rounds of iterative retrieval. Further analyses show that structured event-centric evidence provides a more effective memory interface than raw frames or flattened captions, and that construction-time unification outperforms retrieval-time fusion. Together, these results suggest that organizing multimodal memory around events provides a more natural and effective foundation for grounded, attributable long-video QA.

这种以事件为中心的设计，在准确率、效率和证据归因方面带来了稳定提升。 在 EgoLifeQA、Ego-R1 Bench 和 Video-MME (L) 三个长视频问答基准上，EM²Mem 相较最强记忆基线的平均准确率分别提高了 2.0%、2.4% 和 3.7%，同时单次查询延迟降低至原来的 1/4.67，推理 token 总量减少了 63.66%。 它也能更精确地定位证据：严格事件级 Top-5 召回率比 WorldMM 进行五轮迭代检索时高出 7.0 个百分点。 进一步分析表明，与原始帧或扁平化的视频描述相比，以事件为中心的结构化证据提供了更有效的记忆接口，且构建阶段的统一优于检索阶段的融合。 综合来看，这些结果表明，围绕事件组织多模态记忆，能够为有证据依据、可归因的长视频问答提供更自然且更有效的基础。

2. Background

Given a long video with visual and text-form evidence streams $V=\{V^{\mathrm{vis}},V^{\mathrm{text}}\}$ and a question $q$, long-video QA aims to generate an answer $\hat{y}$ grounded in video-specific evidence. Here, $V^{\mathrm{text}}$ includes transcripts, OCR, subtitles, dialogue, and other textual cues. Since relevant evidence is often sparse, temporally distant, and distributed across modalities, directly feeding the full video into an LLM or MLLM is inefficient and may obscure cross-modal dependencies.

给定包含视觉与文本形式证据流 $V=\{V^{\mathrm{vis}},V^{\mathrm{text}}\}$ 的长视频以及问题 $q$，长视频问答旨在生成一个以该视频中特定证据为依据的答案 $\hat{y}$。 其中，$V^{\mathrm{text}}$ 包含转录文本、OCR、字幕、对话及其他文本线索。 由于相关证据往往稀疏、在时间上相距较远且分布于不同模态中，直接将完整视频输入 LLM 或多模态大语言模型（MLLM）效率低下，还可能掩盖跨模态依赖关系。

Existing multimodal memory systems typically convert long videos or agent experiences into searchable modality-typed units:

现有多模态记忆系统通常将长视频或智能体经历转换为按模态区分的可检索单元：

$$\begin{array}{}\text{(1)}& \begin{array}{rl}{u}_j^m& =(s_j^m,e_j^m,p_j^m,{\tau }_j,m,{\ell }_j),\\ & m\in \{\mathrm{vis},\mathrm{text}\},\\ s_j^{\mathrm{vis}}& =c_j^{\mathrm{clip}}={\varphi }_{\mathrm{cap}}(F_j,K_j),\\ s_j^{\mathrm{text}}& ={\varphi }_{\mathrm{text}}(r_j,o_j,d_j).\end{array}\end{array}$$

Here, $s_j^m$ and $e_j^m$ denote the searchable summary and embedding, $p_j^m$ points to the source evidence, ${\tau }_j$ is the timestamp, and ${\ell }_j$ stores links to other units. Visual frames $F_j$ and keyframes $K_j$ are converted into clip captions, while transcripts $r_j$, OCR/subtitles $o_j$, and other textual cues $d_j$ are normalized into text summaries. Although such units provide a unified storage interface, they remain organized around individual modality-specific signals or compressed observations rather than coherent multimodal events.

这里，$s_j^m$ 和 $e_j^m$ 分别表示可检索摘要和嵌入，$p_j^m$ 指向源证据，${\tau }_j$ 为时间戳，${\ell }_j$ 存储指向其他单元的链接。 视觉帧 $F_j$ 和关键帧 $K_j$ 被转换为片段描述，而转录文本 $r_j$、OCR／字幕 $o_j$ 以及其他文本线索 $d_j$ 被规范化为文本摘要。 尽管这些单元提供了统一的存储接口，但它们仍围绕单一模态信号或压缩后的观测进行组织，而不是围绕连贯的多模态事件。

3. Method

图2：EM²Mem 概览。EM²Mem 将长视频解析为事件锚点，构建包含局部记录 $R_i$ 和多尺度上下文视图 ${\mathcal{C}}_i$ 的事件中心多模态记忆单元，并通过情景图 $G_E$ 和语义图 $G_S$ 将它们连接起来。给定问题后，轻量检索会选择并扩展相关事件单元，汇编证据以生成答案。

图2：EM²Mem 概览。EM²Mem 将长视频解析为事件锚点，构建包含局部记录 $R_i$ 和多尺度上下文视图 ${\mathcal{C}}_i$ 的事件中心多模态记忆单元，并通过情景图 $G_E$ 和语义图 $G_S$ 将它们连接起来。给定问题后，轻量检索会选择并扩展相关事件单元，汇编证据以生成答案。

3.1. Overview

We propose EM²Mem, an Event-Centric Multimodal Memory framework for long-video reasoning with LLMs. EM²Mem follows an align-then-retrieve design for long-video memory, as illustrated in Figure 2. It first parses a long video into short temporal events and uses event anchors as shared indices to align heterogeneous evidence during memory construction, rather than retrieving isolated modality-specific fragments at inference time. To support reasoning beyond individual events, EM²Mem further builds lightweight event-linked graph memory: an episodic graph captures concrete cross-event relations, including shared entities, objects, scenes, topics, and temporal transitions, while a semantic graph stores higher-level regularities grounded in supporting events. At inference time, EM²Mem retrieves relevant event cells, expands them with graph-linked evidence, and compiles a compact query-specific evidence view for answer generation.

我们提出 EM²Mem，一个使用 LLM 进行长视频推理的 事件中心多模态记忆 框架。 如 图2 所示，EM²Mem 对长视频记忆采用 先对齐、后检索 的设计。 它首先将长视频解析为较短的时间事件，使用事件锚点作为共享索引，在记忆构建阶段对齐异构证据，而不是在推理时检索特定模态的孤立片段。 为支持超越单个事件的推理，EM²Mem 进一步构建了轻量的事件关联图记忆：情景图捕捉具体的跨事件关系，包括共享实体、对象、场景、主题及时间转移；语义图则存储以支撑事件为依据的高层规律。 在推理阶段，EM²Mem 检索相关事件单元，利用图关联证据扩展这些单元，并汇编一个针对查询的紧凑证据视图，用于生成答案。

3.2. Event-Centric Multimodal Memory Schema

Instead of treating a video as a flat sequence of frames or captions, EM²Mem represents it as an event-indexed memory. We divide the video into short base segments, e.g. 30-second clips, and associate each segment with an event anchor:

EM²Mem 将视频表示为由事件索引的记忆，而不是将其视为扁平的帧序列或描述序列。 我们将视频划分为较短的基础片段，例如 30 秒的片段，并为每个片段关联一个事件锚点：

$$\begin{array}{}\text{(2)}& e_i=(i,{\tau }_i^s,{\tau }_i^e).\end{array}$$

An event anchor is a temporal address rather than memory content. It provides a shared indexing key for heterogeneous evidence and higher-level abstractions. Formally, EM²Mem represents the video as a set of Event-Centric Multimodal Memory Cells:

事件锚点是一个时间地址，而非记忆内容。 它为异构证据和更高层的抽象提供共享索引键。 形式上，EM²Mem 将视频表示为一组 事件中心多模态记忆单元：

$$\begin{array}{}\text{(3)}& \begin{array}{rl}{\mathcal{M}}_{{\mathrm{EM}}^2\mathrm{Mem}}& =(\{{\mu }_i{\}}_{i=1}^N,G_E,G_S),\\ {\mu }_i& =(e_i,R_i,{\mathcal{C}}_i).\end{array}\end{array}$$

Each event memory cell ${\mu }_i$ is anchored at $e_i$ and stores local multimodal evidence $R_i$ and multi-scale temporal context views ${\mathcal{C}}_i$. Beyond cell-level memory, EM²Mem builds two global event-linked graphs: $G_E$ captures cross-event episodic relations and temporal transitions, while $G_S$ captures long-term semantic relations.

每个事件记忆单元 ${\mu }_i$ 都以 $e_i$ 为锚点，存储局部多模态证据 $R_i$ 和多尺度时间上下文视图 ${\mathcal{C}}_i$。 除单元级记忆外，EM²Mem 还构建两个全局的事件关联图：$G_E$ 捕捉跨事件情景关系和时间转移，$G_S$ 则捕捉长期语义关系。

To make this schema concrete, suppose an event segment shows a person discussing a flower-based craft plan while interacting with flowers and vases. The corresponding memory cell binds the dialogue context, keyframe captions, and structured fields such as ${\mathrm{Act}}_i=\mathit{\text{discussing a craft plan}}$, ${\mathrm{Obj}}_i=\{\mathit{\text{flowers}},\mathit{\text{vases}}\}$, and ${\mathrm{Top}}_i=\mathit{\text{flower-based craft}}$ under the same event anchor $e_i$, rather than storing them as separate captions, transcript snippets, and frames. Its context views ${\mathcal{C}}_i$ link the event to surrounding temporal summaries, while the graphs connect it to related entities, objects, topics, and recurring patterns. This design makes the memory cell query-ready: questions about the plan, involved objects, or nearby activities can retrieve one grounded event cell and its linked context instead of stitching together isolated modality-specific fragments at inference time. All evidence is grounded to event anchors via $\alpha (\cdot )$, preserving heterogeneous memory forms within a event-centric schema.

为具体说明这一模式，假设某个事件片段中，一名参与者一边摆弄花卉和花瓶，一边讨论以花卉为主题的手工制作计划。 对应的记忆单元会将对话上下文、关键帧描述，以及 ${\mathrm{Act}}_i=\mathit{\text{discussing a craft plan}}$、${\mathrm{Obj}}_i=\{\mathit{\text{flowers}},\mathit{\text{vases}}\}$ 和 ${\mathrm{Top}}_i=\mathit{\text{flower-based craft}}$ 等结构化字段绑定到同一个事件锚点 $e_i$ 下，而不是将它们存储为彼此分离的描述、转录片段和帧。 其上下文视图 ${\mathcal{C}}_i$ 将事件连接到周围时间段的摘要，而图结构则将其连接到相关实体、对象、主题和反复出现的模式。 这种设计使记忆单元能够直接服务于查询：关于计划、涉及的对象或相邻活动的问题，可以检索一个有证据依据的事件单元及其关联上下文，而无需在推理阶段拼接特定模态的孤立片段。 所有证据都通过 $\alpha (\cdot )$ 落实到事件锚点，在以事件为中心的模式中保留异构记忆形式。

3.3. Multimodal Memory Construction

Multimodal Event Record. For each base event segment, EM²Mem constructs a multimodal event record that stores the local evidence observed within the event:

多模态事件记录。 对于每个基础事件片段，EM²Mem 都构建一条多模态事件记录，存储该事件内观测到的局部证据：

$$\begin{array}{}\text{(4)}& \begin{array}{rl}{R}_i& =(c_i,r_i,k_i,z_i,{\tau }_i),\\ z_i& =({\mathrm{Act}}_i,{\mathrm{Obj}}_i,{\mathrm{Top}}_i,\\ & {\mathrm{Scen}}_i,{\mathrm{Ent}}_i).\end{array}\end{array}$$

Here, $c_i$ is a visual keyframe caption, $r_i$ is the transcript or dialogue context, $k_i$ denotes representative keyframes, and $z_i$ contains structured metadata, including actions, objects, dialogue topics, visual scenes, and visual entities. These fields are obtained through multimodal parsing modules, including captioning, transcript alignment, keyframe selection, and structured field extraction.

这里，$c_i$ 为视觉关键帧描述，$r_i$ 为转录文本或对话上下文，$k_i$ 表示代表性关键帧，$z_i$ 包含结构化元数据，包括动作、对象、对话主题、视觉场景和视觉实体。 这些字段由多模态解析模块获得，涉及描述生成、转录文本对齐、关键帧选择和结构化字段提取。

Temporal Context Views. To support reasoning at different temporal granularities, EM²Mem further builds temporal context views over multiple scales, such as 3 minutes, 10 minutes, and 1 hour. At each scale, consecutive event anchors are grouped into a temporal block and summarized into a context view:

时间上下文视图。 为支持不同时间粒度的推理，EM²Mem 进一步在多个尺度上构建时间上下文视图，例如 3 分钟、10 分钟和 1 小时。 在每个尺度上，将连续的事件锚点分组为一个时间块，并概括为上下文视图：

$$\begin{array}{}\text{(5)}& h_j^{(l)}=(u_{j,l}^{\mathrm{text}},u_{j,l}^{\mathrm{vis}},m_{j,l},{\tau }_{j,l},l).\end{array}$$

The textual summary $u_{j,l}^{\mathrm{text}}$ captures narrative and dialogue context, the visual summary $u_{j,l}^{\mathrm{vis}}$ summarizes visual evidence, and $m_{j,l}$ aggregates normalized metadata such as actions, objects, scenes, and topics. Each context view is anchored to all base events covered by its temporal span, and the context-view set ${\mathcal{C}}_i$ of an event cell contains all views whose spans include $e_i$.

文本摘要 $u_{j,l}^{\mathrm{text}}$ 捕捉叙事和对话上下文，视觉摘要 $u_{j,l}^{\mathrm{vis}}$ 概括视觉证据，$m_{j,l}$ 则聚合动作、对象、场景和主题等规范化元数据。 每个上下文视图都锚定到其时间跨度覆盖的全部基础事件；一个事件单元的上下文视图集合 ${\mathcal{C}}_i$ 包含所有时间跨度覆盖 $e_i$ 的视图。

3.4. Event-Linked Memory Graph Construction

Beyond event memory cells, EM²Mem builds two lightweight graph indices over shared event anchors: an episodic graph $G_E$ and a semantic graph $G_S$. These graphs are used as auxiliary indices rather than as new graph formalisms. The episodic graph connects event anchors with typed entities such as people, objects, locations, scenes, and topics, and also includes temporal transitions between adjacent events. Each relation is grounded to its supporting event anchors, allowing graph-derived clues to be traced back to concrete video evidence.

除事件记忆单元外，EM²Mem 还在共享事件锚点之上构建两个轻量图索引：情景图 $G_E$ 和语义图 $G_S$。 这些图用作辅助索引，并非新的图形式体系。 情景图将事件锚点与人物、对象、地点、场景和主题等带类型的实体连接起来，同时也包含相邻事件之间的时间转移。 每个关系都落实到支撑它的事件锚点，使从图中得到的线索能够追溯到具体的视频证据。

The semantic graph summarizes recurring patterns from episodic evidence, such as habits, preferences, routines, and stable relationships. Unlike local event records, which describe concrete observations, semantic facts capture long-term regularities that may span multiple distant events. These facts are also linked back to their supporting anchors, so that high-level knowledge remains evidence-grounded. Together, $G_E$ and $G_S$ complement event memory cells by providing cross-event connectivity and long-term semantic context.

语义图从情景证据中概括反复出现的模式，例如习惯、偏好、日常规律和稳定关系。 局部事件记录描述的是具体观测，而语义事实捕捉的是可能跨越多个遥远事件的长期规律。 这些事实也会回链到支撑它们的锚点，使高层知识始终具有证据依据。 $G_E$ 与 $G_S$ 共同提供跨事件连接和长期语义上下文，作为事件记忆单元的补充。

3.5. Memory Retrieval and Ranking

At inference time, EM²Mem performs lightweight event-level readout over pre-aligned memory cells, which shifts retrieval from query-time cross-modal fusion to compact event-cell selection and expansion. Given a question $q$, it first identifies relevant event memory cells using signals from local multimodal records, multi-scale temporal context views, and graph-linked evidence. It then performs lightweight expansion through temporal transitions, shared entities, objects, locations, topics, and relevant semantic facts, so that complementary evidence from nearby or semantically related events can be included.

在推理阶段，EM²Mem 对预先对齐的记忆单元执行轻量的事件级读取，将检索从查询时的跨模态融合转变为紧凑的事件单元选择与扩展。 给定问题 $q$，它首先利用局部多模态记录、多尺度时间上下文视图以及图关联证据中的信号，识别相关事件记忆单元。 随后，通过时间转移、共享实体、对象、地点、主题和相关语义事实进行轻量扩展，以纳入邻近事件或语义相关事件中的互补证据。

An LLM-based selector further filters the candidate event cells and keeps only the evidence needed to answer the question. For each selected event, EM²Mem compiles a query-specific evidence view $E_q$, including relevant captions, transcripts, structured visual fields, temporal summaries, semantic facts, and selected keyframes. The final answer $\hat{y}$ is generated by ${\mathrm{LLM}}_{\mathrm{ans}}$ conditioned on the question $q$ and the compact evidence view $E_q$.

基于 LLM 的选择器进一步筛选候选事件单元，只保留回答问题所需的证据。 对于每个选中的事件，EM²Mem 汇编一个针对查询的证据视图 $E_q$，其中包含相关描述、转录文本、结构化视觉字段、时间摘要、语义事实和选定的关键帧。 最终答案 $\hat{y}$ 由 ${\mathrm{LLM}}_{\mathrm{ans}}$ 在问题 $q$ 和紧凑证据视图 $E_q$ 的条件下生成。

4. Experiment

4.1. Experimental Setup

Datasets and Metrics. We evaluate EM²Mem on three multiple-choice long-video QA benchmarks: EgoLifeQA, Ego-R1 Bench, and Video-MME (L). Together, they cover week-long egocentric daily-life QA, ultra-long egocentric reasoning, and open-domain long-video understanding. Following the standard protocol of each benchmark, we report category-level and average accuracy. Dataset statistics, category definitions, and license details are provided in Appendix A.

数据集与指标。 我们在 EgoLifeQA、Ego-R1 Bench 和 Video-MME (L) 三个多项选择长视频问答基准上评估 EM²Mem。 它们共同覆盖了为期一周的第一人称日常生活问答、超长第一人称视频推理和开放域长视频理解。 按照各基准的标准协议，我们报告各类别准确率和平均准确率。 数据集统计信息、类别定义和许可详情见 附录 A。

Baselines. We compare EM²Mem with representative baselines spanning base MLLMs, long-video MLLMs, RAG-based video QA methods, and memory-based long-video reasoning frameworks, as listed in Tables 1 and 3. For the strongest and most related memory baseline, we report both the originally published WorldMM results and our reproduced WorldMM${}^{†}$ results under the same evaluation setting as EM²Mem. Detailed baseline configurations are provided in Appendix B.1.

基线。 我们将 EM²Mem 与具有代表性的基线进行比较，涵盖基础 MLLM、长视频 MLLM、基于 RAG 的视频问答方法，以及基于记忆的长视频推理框架，具体列于 表1 和 表3。 对于最强且最相关的记忆基线，我们同时报告 WorldMM 原论文发表的结果，以及在与 EM²Mem 相同评测设置下复现的 WorldMM${}^{†}$ 结果。 详细基线配置见 附录 B.1。

4.2. Main Results

表1：EgoLifeQA 和 Ego-R1 Bench 上的性能比较。WorldMM 表示原论文报告的结果。† 表示我们在与 EM²Mem 相同的评测设置下复现的结果。对于 Ego-R1 Bench，我们报告已发表的 WorldMM 结果，因为复现 WorldMM 需要为全部六名参与者重建个体级长视频记忆，会产生大量构建和推理成本；相应条目以“–”标记。

Model	EgoLifeQA						Ego-R1 Bench
	Ent.	EvR.	Hab.	Rel.	Task	Avg.	Ent.	EvR.	Hab.	Rel.	Task	Avg.
Base Models
Qwen3-VL-8B	35.2	30.2	39.3	46.4	46.0	38.6	31.8	41.5	38.5	42.1	44.7	35.7
Gemini 2.5 Pro	43.2	40.5	41.0	55.2	52.4	46.4	43.9	56.1	53.9	47.4	47.4	46.7
GPT-5	47.2	42.1	47.5	53.6	55.6	48.6	41.8	58.5	53.9	52.6	50.0	46.3
Long Video LLMs
VideoChat-Flash	28.8	32.5	37.7	37.6	38.1	34.2	43.4	43.9	38.5	31.6	44.7	42.7
Time-R1	39.2	50.8	65.6	48.8	47.6	48.8	49.2	48.8	46.2	42.1	44.7	48.0
Video-RTS	40.8	48.4	62.3	48.8	47.6	48.2	47.6	46.3	53.9	52.6	47.4	48.0
RAG-based Video LLMs
LightRAG	40.8	48.4	67.2	50.4	44.4	48.8	54.0	61.0	46.2	42.1	42.1	52.3
HippoRAG	48.8	60.3	70.5	60.8	66.7	59.6	54.5	65.9	69.2	52.6	50.0	56.0
Video-RAG	49.6	56.3	67.2	55.2	54.0	55.4	48.7	58.5	53.9	47.4	44.7	49.7
Memory-based Video LLMs
EgoRAG	40.0	56.3	62.3	54.4	52.4	52.0	46.6	56.1	46.2	47.4	55.3	49.0
Ego-R1	51.2	53.2	63.9	50.4	50.8	53.0	50.8	63.4	38.5	36.8	57.9	52.0
HippoMM	45.6	53.2	70.5	55.2	58.7	54.6	51.9	56.1	46.2	52.6	57.9	53.0
M3-Agent	44.4	54.8	62.3	56.8	54.0	53.5	52.4	58.5	38.5	42.1	52.6	52.0
WorldMM	62.4	64.3	75.4	62.4	71.4	65.6	64.6	70.7	76.9	57.9	63.2	65.3
WorldMM†	57.6	65.1	68.9	68.8	60.3	64.0	–
EM2Mem (Ours)	60.8	61.1	63.9	72.8	74.6	66.0	74.6	53.7	69.2	47.4	57.9	67.7

表1：EgoLifeQA 和 Ego-R1 Bench 上的性能比较。WorldMM 表示原论文报告的结果。† 表示我们在与 EM²Mem 相同的评测设置下复现的结果。对于 Ego-R1 Bench，我们报告已发表的 WorldMM 结果，因为复现 WorldMM 需要为全部六名参与者重建个体级长视频记忆，会产生大量构建和推理成本；相应条目以“–”标记。

表2：EgoLifeQA 上的推理效率比较。延迟按单次查询测量；实际运行时间遵循各方法的实际评测流程。

Metric	WorldMM	EM2Mem	Gain
Per-query inference efficiency
Avg. latency / query (s)	459.00	98.21	4.67×
End-to-end evaluation throughput
Wall-clock time (s)	229,502	6,138	37.39×
Inference token cost
Input tokens	31.95M	13.29M	58.42%↓
Output tokens	10.08M	1.99M	80.29%↓
Total tokens	42.03M	15.27M	63.66%↓

表2：EgoLifeQA 上的推理效率比较。延迟按单次查询测量；实际运行时间遵循各方法的实际评测流程。

表3：Video-MME (L) 上各类别的性能。WorldMM 表示原论文报告的结果。† 表示我们在与 EM²Mem 相同的评测设置下复现的结果。

Model	ARES	AREC	ATTR	CNT	ISYN	OCR	ORES	OREC	SPER	SRES	TPER	TRES	Avg.
Base Models
Qwen3-VL-8B	62.2	54.0	51.9	43.8	68.1	42.9	62.9	57.4	33.3	45.5	33.3	67.0	61.0
Gemini 2.5 Pro	56.9	47.6	66.7	41.7	71.8	57.1	53.3	40.7	0.0	72.7	66.7	48.4	55.7
GPT-5	71.1	69.8	70.4	47.9	88.3	57.1	75.8	74.1	33.3	72.7	50.0	75.8	74.3
Long Video LLMs
VideoChat-Flash	35.0	42.9	37.0	31.3	34.4	42.9	60.0	46.3	33.3	54.5	33.3	46.2	44.1
Time-R1	20.6	28.6	25.9	35.4	31.9	35.7	53.3	48.2	33.3	36.4	50.0	44.0	37.6
Video-RTS	43.3	52.4	40.7	39.6	33.7	42.9	60.8	53.7	33.3	45.5	50.0	49.5	47.9
RAG-based Video LLMs
LightRAG	41.7	30.2	40.7	35.4	54.0	50.0	46.7	61.1	33.3	45.5	50.0	52.8	46.6
HippoRAG	45.6	47.6	40.7	37.5	52.2	42.9	52.9	64.8	66.7	54.5	50.0	70.3	52.1
Video-RAG	51.7	47.6	37.0	39.6	49.7	57.1	62.1	68.5	66.7	45.5	50.0	68.1	55.4
Memory-based Video LLMs
EgoRAG	31.1	55.6	33.3	22.9	41.1	28.6	44.6	48.2	33.3	54.5	66.7	48.4	41.1
Ego-R1	37.2	52.4	40.7	35.4	38.0	35.7	42.1	51.9	66.7	63.6	50.0	52.8	42.7
HippoMM	41.1	42.9	55.6	35.4	38.7	35.7	37.9	53.7	33.3	54.5	50.0	47.3	41.6
M3-Agent	52.2	57.1	59.3	45.8	51.5	42.9	54.6	64.8	33.3	45.5	50.0	71.4	55.3
WorldMM	81.1	73.0	70.4	54.2	85.3	42.9	75.0	77.8	33.3	72.7	66.7	79.1	76.6
WorldMM†	73.3	68.3	77.8	60.4	80.2	50.0	72.4	77.8	33.3	90.9	66.7	71.1	73.1
EM2Mem (Ours)	77.2	76.2	77.8	64.6	80.7	64.3	77.0	77.8	33.3	81.8	50.0	79.1	76.8

表3：Video-MME (L) 上各类别的性能。WorldMM 表示原论文报告的结果。† 表示我们在与 EM²Mem 相同的评测设置下复现的结果。

Accuracy across long-video QA benchmarks. Tables 1 and 3 report the main results on EgoLifeQA, Ego-R1 Bench, and Video-MME (L). Overall, EM²Mem achieves strong performance against base MLLMs, long-video LLMs, RAG-based methods, and memory-based video reasoning systems. We report both previously published WorldMM results and our reproduced WorldMM${}^{†}$ results for reference; since reproduced results are obtained under the same evaluation setting as EM²Mem, they provide the most controlled comparison. Under this setting, EM²Mem improves over WorldMM${}^{†}$ by 2.0 points on EgoLifeQA (66.0 vs. 64.0), and 3.7 points on Video-MME (L) (76.8 vs. 73.1). On Ego-R1 Bench, where we use the originally reported WorldMM result for comparison, EM²Mem achieves 67.7 average accuracy, outperforming WorldMM by 2.4 points (67.7 vs. 65.3). These gains are especially visible on RelationMap and TaskMaster in EgoLifeQA, EntityLog in Ego-R1 Bench, and AREC, CNT, OCR, and ORES in Video-MME (L), suggesting that event-indexed retrieval units are useful for grounded entity tracking, task-state reasoning, and cross-modal evidence aggregation. Compared with the originally reported WorldMM numbers, EM²Mem remains competitive and slightly higher on average, although WorldMM is stronger on several habit, temporal, and synthetic reasoning categories.

不同长视频问答基准上的准确率。 表1 和 表3 报告了 EgoLifeQA、Ego-R1 Bench 和 Video-MME (L) 上的主要结果。 总体而言，与基础 MLLM、长视频 LLM、基于 RAG 的方法以及基于记忆的视频推理系统相比，EM²Mem 表现出较强的性能。 我们同时报告此前发表的 WorldMM 结果和我们复现的 WorldMM${}^{†}$ 结果作为参考；由于复现结果是在与 EM²Mem 相同的评测设置下获得的，因此能提供控制条件最一致的比较。 在这一设置下，EM²Mem 在 EgoLifeQA 上比 WorldMM${}^{†}$ 提高了 2.0 个百分点（66.0 对比 64.0），在 Video-MME (L) 上提高了 3.7 个百分点（76.8 对比 73.1）。 在 Ego-R1 Bench 上，我们使用 WorldMM 原论文报告的结果进行比较，EM²Mem 的平均准确率为 67.7，比 WorldMM 高出 2.4 个百分点（67.7 对比 65.3）。 这些提升在 EgoLifeQA 的 RelationMap 和 TaskMaster、Ego-R1 Bench 的 EntityLog，以及 Video-MME (L) 的 AREC、CNT、OCR 和 ORES 类别上尤其明显，表明由事件索引的检索单元有助于有证据依据的实体追踪、任务状态推理和跨模态证据聚合。 相较 WorldMM 原论文报告的数值，EM²Mem 仍具有竞争力，平均表现也略高，尽管 WorldMM 在若干习惯、时间及综合推理类别上更强。

Inference efficiency and scalability. Beyond accuracy, Table 2 compares EM²Mem with WorldMM along three efficiency dimensions: per-query latency, end-to-end wall-clock time, and inference token cost. We report both latency and wall-clock time because they capture different aspects of efficiency: latency measures the delay of an individual request, while wall-clock time reflects end-to-end throughput under each method's practical execution pipeline. In this setting, EM²Mem processes independent questions with 8 workers, whereas WorldMM is constrained by incremental HippoRAG graph construction during inference. EM²Mem reduces average latency from 459.00s to 98.21s, yielding a $4.67\times$ per-query speedup. This shows that the efficiency gain is not only due to parallel execution. The main reason is that EM²Mem reads from pre-constructed event-indexed memory cells instead of performing evidence alignment and graph organization during inference. EM²Mem further reduces wall-clock evaluation time from 229,502s to 6,138s, corresponding to a $37.39\times$ throughput improvement, and lowers total inference token cost from 42.03M to 15.27M, a 63.66% reduction. These results indicate that construction-time memory organization improves not only answer quality, but also the inference-time cost of evidence readout. Figure 3e further contextualizes these savings by amortizing the offline construction cost over repeated queries on the same memory. Although EM²Mem incurs a higher upfront construction cost, its lower inference-time wall-clock cost reaches break-even after roughly 23–24 queries, while token usage requires a larger reuse scale to break even; we provide the full end-to-end derivation in Appendix C.2. Taken together, these results suggest that event-indexed retrieval units improve long-video QA accuracy while making inference-time evidence readout more scalable.

推理效率与可扩展性。 除准确率之外，表2 从单次查询延迟、端到端实际运行时间和推理 token 成本三个效率维度，对 EM²Mem 与 WorldMM 进行比较。 我们同时报告延迟和实际运行时间，因为二者反映效率的不同方面：延迟衡量单个请求的等待时间，而实际运行时间反映各方法实际执行流程下的端到端吞吐量。 在此设置下，EM²Mem 使用 8 个工作进程处理相互独立的问题，而 WorldMM 则受到推理阶段增量构建 HippoRAG 图的限制。 EM²Mem 将平均延迟从 459.00 秒降至 98.21 秒，获得 $4.67\times$ 的单次查询加速。 这表明效率提升并非仅来自并行执行。 主要原因在于，EM²Mem 从预先构建的事件索引记忆单元中读取信息，而不是在推理阶段进行证据对齐和图组织。 EM²Mem 进一步将评测的实际运行时间从 229,502 秒降至 6,138 秒，对应 $37.39\times$ 的吞吐量提升，并将推理 token 总成本从 42.03M 降至 15.27M，减少了 63.66%。 这些结果说明，在构建阶段组织记忆，不仅改善了答案质量，也降低了推理阶段读取证据的成本。 图3e 通过将离线构建成本摊销到同一记忆上的重复查询中，进一步展示了这些节省的意义。 尽管 EM²Mem 的前期构建成本较高，但由于推理阶段的实际运行时间成本更低，大约在 23–24 次查询后即可达到盈亏平衡；token 用量则需要更大的复用规模才能达到盈亏平衡，完整的端到端推导见 附录 C.2。 综合来看，这些结果表明，由事件索引的检索单元既提高了长视频问答准确率，也使推理阶段的证据读取更具可扩展性。

4.3. Ablation Study

Table 4 and Figure 3b show the overall ablation results on EgoLifeQA. Removing temporal context views leads to the largest drop, decreasing accuracy from 66.0% to 60.4%. Removing semantic facts and episodic graph support also substantially hurts performance, reducing accuracy to 61.4% and 61.6%, respectively. These results validate the design of combining local event records, temporal context views, episodic graph, and semantic facts in an event-indexed memory.

表4 和 图3b 展示了 EgoLifeQA 上的总体消融结果。 移除时间上下文视图造成的性能下降最大，准确率从 66.0% 降至 60.4%。 移除语义事实和情景图支持也会显著损害性能，使准确率分别降至 61.4% 和 61.6%。 这些结果验证了在事件索引记忆中结合局部事件记录、时间上下文视图、情景图和语义事实这一设计的有效性。

表4：EgoLifeQA 上的总体消融结果。$\mathrm{\Delta }$ 表示相较完整 EM²Mem 模型的准确率绝对下降值。

Method	Overall Acc.	Δ
EM2Mem	66.0	–
w/o Temporal Context Views	60.4	−5.6
w/o Semantic Memory	61.4	−4.6
w/o Episodic Graph	61.6	−4.4
Local 30s Event Record	64.0	−2.0

表4：EgoLifeQA 上的总体消融结果。$\mathrm{\Delta }$ 表示相较完整 EM²Mem 模型的准确率绝对下降值。

Interestingly, the local 30-second record retrieval baseline remains competitive, achieving 64.0%. This suggests that fine-grained local event record already provide strong evidence for short-range factual questions. However, the full EM²Mem still achieves the best overall accuracy. The category-wise results in Table 11 at Appendix C.3 show that the advantage of EM²Mem is more pronounced on relation- and task-oriented questions, where evidence often needs to be aggregated across temporally distant events and connected through entity, object, or activity relations.

有趣的是，局部 30 秒记录检索基线仍具有竞争力，达到 64.0%。 这表明，细粒度局部事件记录已经能够为短时间范围内的事实性问题提供有力证据。 不过，完整的 EM²Mem 仍取得了最佳总体准确率。 附录 C.3 中 表11 的分类型结果表明，EM²Mem 在关系导向和任务导向的问题上优势更为明显；这类问题往往需要跨越时间上相距较远的事件聚合证据，并通过实体、对象或活动关系将它们联系起来。

4.4. Analysis

Event binding turns heterogeneous evidence into answerable memory units. We analyze how the representation and fusion stage of multimodal evidence affect memory-based long-video QA, using the first 250 questions of EgoLifeQA. Our analysis spans two dimensions: (1) memory representation of visual evidence, including raw frames, caption-level descriptions, and structured event fields; and (2) unification stage, comparing retrieval-time fusion, where independent stores are combined during inference, with construction-time unification, where cross-modal evidence is aligned under shared event anchors before indexing.

事件绑定将异构证据转化为能够支持作答的记忆单元。 我们使用 EgoLifeQA 的前 250 个问题，分析多模态证据的表示形式和融合阶段如何影响基于记忆的长视频问答。 分析涵盖两个维度：（1）视觉证据的记忆表示，包括原始帧、描述级表示和结构化事件字段；（2）统一阶段，比较在推理时组合独立存储的检索阶段融合，与在索引前便通过共享事件锚点对齐跨模态证据的构建阶段统一。

Structured event fields bridge multimodal evidence and natural-language questions. As shown in Table 5 and Figure 3a, the representation of visual evidence has a substantial impact on QA performance. Across both fusion stages, structured evidence outperforms raw and caption-level memories, achieving 71.2% accuracy when combined with construction-time unification. This suggests typed visual fields form a more effective memory interface for LLM-based QA: compared with raw frames, they are easier to retrieve, verbalize, and attribute; compared with flattened captions, they preserve explicit entities, actions, scenes, and other question-relevant cues. Rather than serving only as a visual representation, structured evidence acts as an answerable interface between multimodal observations and natural-language questions.

结构化事件字段连接多模态证据与自然语言问题。 如 表5 和 图3a 所示，视觉证据的表示形式会显著影响问答性能。 在两种融合阶段下，结构化证据均优于原始记忆和描述级记忆，与构建阶段统一结合时，准确率达到 71.2%。 这表明，带类型的视觉字段为基于 LLM 的问答提供了更有效的记忆接口：与原始帧相比，它们更容易被检索、用语言表达和归因；与扁平化描述相比，它们保留了显式的实体、动作、场景及其他与问题相关的线索。 结构化证据不仅是一种视觉表示，更充当了连接多模态观测与自然语言问题、能够支持作答的接口。

表5：EgoLifeQA 前 250 个问题上的准确率（%）。各行采用不同的视觉证据记忆表示；各列采用不同的多模态证据统一阶段。

Memory Representation	Unification Stage
	Retrieval	Construction
Raw frames	66.8	68.0
Flattened captions	65.2	65.6
Structured event fields	68.0	71.2

表5：EgoLifeQA 前 250 个问题上的准确率（%）。各行采用不同的视觉证据记忆表示；各列采用不同的多模态证据统一阶段。

Construction-time unification mitigates retrieve-then-align bottlenecks. Table 5 and Figure 3a also show that construction-time unification consistently improves over retrieval-time fusion. Retrieval-time fusion follows a retrieve-then-align pattern: each modality is retrieved independently, so evidence that is weakly relevant in isolation may be discarded before it can be combined with complementary cues from other modalities. In contrast, construction-time unification aligns captions, transcripts, and structured visual fields into shared event-level retrieval units before indexing. This makes retrieved evidence more semantically coherent, temporally grounded, and generation-ready. It also preserves evidence whose relevance emerges only after cross-modal binding, such as an object mentioned visually but disambiguated by nearby dialogue, or an action described in text but grounded by keyframes. The gain is largest for structured evidence (+3.2%), suggesting that explicit fields such as entities and actions provide useful anchors for cross-modal alignment. Together, these results support our central claim: effective multimodal memory should unify heterogeneous evidence during memory construction, producing event-level retrieval units that are directly usable for grounded and attributable long-video QA.

构建阶段统一缓解了先检索、后对齐的瓶颈。 表5 和 图3a 还表明，构建阶段统一始终优于检索阶段融合。 检索阶段融合遵循先检索、后对齐的模式：各模态被独立检索，因此单独看来相关性较弱的证据，可能尚未与其他模态的互补线索结合就已被丢弃。 相比之下，构建阶段统一在索引之前，就将描述、转录文本和结构化视觉字段对齐到共享的事件级检索单元中。 这使检索到的证据在语义上更连贯、在时间上更有依据，也能够直接服务于生成。 它还保留了那些只有在跨模态绑定后才显现相关性的证据，例如视觉中出现的对象需要借助邻近对话消歧，或文本描述的动作需要关键帧提供依据。 结构化证据的增益最大（+3.2%），表明实体、动作等显式字段能够为跨模态对齐提供有用的锚点。 综合来看，这些结果支持了我们的核心主张：有效的多模态记忆应在记忆构建阶段统一异构证据，产生可直接用于有证据依据、可归因的长视频问答的事件级检索单元。

(a) 表示形式与统一时机。

(b) 组件贡献。

(c) 召回率与检索预算。

(d) 准确率与关键帧预算。

(e) 摊销后的实际运行时间与 token 成本。

图3：EM²Mem 设计选择与效率分析。(a) 结构化事件字段结合构建阶段的统一，可获得最高准确率。(b) 逐组件消融显示，各组件对不同问题类型的贡献不同。(c) 随着检索预算增加，EM²Mem 的事件级召回率提高。(d) 视觉关键帧提高了总体准确率，使用三帧时尤其明显。(e) 随着查询次数增加，EM²Mem 的实际运行时间开销逐渐得到摊销；虚线表示实际运行时间和 token 用量的盈亏平衡点。

图3：EM²Mem 设计选择与效率分析。(a) 结构化事件字段结合构建阶段的统一，可获得最高准确率。(b) 逐组件消融显示，各组件对不同问题类型的贡献不同。(c) 随着检索预算增加，EM²Mem 的事件级召回率提高。(d) 视觉关键帧提高了总体准确率，使用三帧时尤其明显。(e) 随着查询次数增加，EM²Mem 的实际运行时间开销逐渐得到摊销；虚线表示实际运行时间和 token 用量的盈亏平衡点。

Keyframes serve as lightweight verification after memory retrieval. We further study how many keyframes should be included in the query-specific evidence view after relevant event cells have been selected. This analysis asks whether event-level textual and structured memory is sufficient, or whether answer generation still benefits from a small amount of visual evidence for verification. As shown in Figure 3d, overall accuracy improves from 63.2% without keyframes to 66.0% with three keyframes. The gains are larger for TaskMaster and RelationMap, where visual evidence helps verify task states, participant identities, and interaction context. By contrast, HabitInsight benefits less consistently because it depends more on repeated long-term behavioral patterns than on local visual verification. Thus, EM²Mem uses compact event-centric multimodal memory as primary evidence and keyframes only for lightweight grounding.

关键帧在记忆检索之后提供轻量验证。 我们进一步研究，在选定相关事件单元后，应当在针对查询的证据视图中包含多少关键帧。 这一分析考察的是，事件级文本和结构化记忆是否已足够，还是答案生成仍能从少量用于验证的视觉证据中获益。 如 图3d 所示，总体准确率从不使用关键帧时的 63.2%，提高到使用三帧时的 66.0%。 TaskMaster 和 RelationMap 的提升更大，因为视觉证据有助于验证任务状态、参与者身份和交互上下文。 相较之下，HabitInsight 的收益不那么稳定，因为它更依赖反复出现的长期行为模式，而非局部视觉验证。 因此，EM²Mem 将紧凑的事件中心多模态记忆用作主要证据，仅用关键帧提供轻量的视觉依据。

表6：严格的 30 秒事件级证据召回率（%）。$\mathrm{\Delta }$ 表示 EM²Mem Top-5 相较 WorldMM 5R 的绝对增益。

Type	WorldMM		EM2Mem		Δ
	1R	5R	Top-1	Top-5
Overall	15.6	23.8	23.0	30.8	+7.0
Ent.	13.6	21.6	20.0	27.2	+5.6
EvR.	19.8	24.6	19.0	23.8	−0.8
Hab.	8.2	23.0	27.9	39.3	+16.3
Rel.	16.0	24.0	21.6	31.2	+7.2
Task.	17.5	27.0	34.9	42.9	+15.9

表6：严格的 30 秒事件级证据召回率（%）。$\mathrm{\Delta }$ 表示 EM²Mem Top-5 相较 WorldMM 5R 的绝对增益。

Event anchors enable single-pass evidence localization. We evaluate strict 30-second event-level recall, where a retrieval is correct only if the retrieved event anchor matches the annotated evidence segment. As shown in Table 6 and Figure 3c, EM²Mem achieves 30.8% Top-5 recall, outperforming WorldMM after five iterative retrieval rounds by 7.0 points. Its Top-1 recall also reaches 23.0%, close to WorldMM 5R at 23.8%, indicating that event-indexed memory can localize relevant evidence with a single-pass retrieval procedure. The gains are largest on HabitInsight and TaskMaster, where evidence often depends on long-term patterns or task states, while EventRecall remains slightly lower, suggesting that localized recall questions may still benefit from iterative caption-level retrieval. We provide additional analysis of selector budgets and a qualitative case study in Appendix C.6 and Appendix C.8.

事件锚点支持单次检索完成证据定位。 我们评估严格的 30 秒事件级召回率，仅当检索到的事件锚点与标注的证据片段匹配时，才视为正确检索。 如 表6 和 图3c 所示，EM²Mem 的 Top-5 召回率达到 30.8%，比 WorldMM 进行五轮迭代检索后高出 7.0 个百分点。 其 Top-1 召回率也达到 23.0%，接近 WorldMM 5R 的 23.8%，表明事件索引记忆能够通过单次检索过程定位相关证据。 HabitInsight 和 TaskMaster 上的增益最大，这两类问题的证据往往依赖长期模式或任务状态；EventRecall 则仍略低，说明局部回忆类问题可能仍会受益于迭代的描述级检索。 我们在 附录 C.6 和 附录 C.8 中提供了关于选择器预算的补充分析以及定性案例研究。

5. Related Work

Long Video Understanding. Video-language research has progressed from video-text pretraining and temporal-aware representation learning to video LLMs that connect visual encoders with LLMs for instruction following and long-form understanding. Recent methods extend video LLMs to longer contexts through visual token compression, sparse or hierarchical memory, long-context adaptation, and temporal reasoning mechanisms. However, as videos scale from minutes to hours or days, relevant evidence becomes sparse, temporally distant, and distributed across modalities, motivating explicit mechanisms for organizing and retrieving video information.

长视频理解。 视频语言研究已从视频文本预训练和时间感知表示学习，发展到将视觉编码器与 LLM 连接起来、用于指令遵循和长篇内容理解的视频 LLM。 近期方法通过视觉 token 压缩、稀疏或分层记忆、长上下文适配以及时间推理机制，将视频 LLM 扩展到更长的上下文。 然而，随着视频从数分钟延长到数小时乃至数天，相关证据变得稀疏、在时间上相距较远且分布于不同模态，因此需要显式组织和检索视频信息的机制。

Multimodal Memory for LLMs. Memory- and retrieval-augmented systems address long-video QA by storing captions, transcripts, keyframes, OCR, clip embeddings, or graph-structured indices and retrieving question-relevant evidence before generation. Closely related egocentric and agentic systems construct hierarchical memories, episode summaries, or adaptive retrieval procedures for long-range reasoning. We provide an extended related work in Appendix D.

面向 LLM 的多模态记忆。 记忆增强和检索增强系统通过存储描述、转录文本、关键帧、OCR、片段嵌入或图结构索引，并在生成前检索与问题相关的证据，来解决长视频问答。 与之密切相关的第一人称和智能体系统，则通过构建分层记忆、情景摘要或自适应检索流程来进行长跨度推理。 更详尽的相关工作见 附录 D。

6. Conclusion

We presented EM²Mem, an event-centric multimodal memory framework that organizes heterogeneous evidence under shared event anchors. By retrieving aligned multimodal events rather than isolated fragments, EM²Mem improves accuracy, evidence localization, and inference efficiency for scalable long-video question answering.

我们提出了 EM²Mem，一个以事件为中心的多模态记忆框架，将异构证据组织在共享事件锚点之下。 通过检索经过对齐的多模态事件，而不是孤立片段，EM²Mem 提高了可扩展长视频问答的准确率、证据定位能力和推理效率。

Limitations

EM²Mem has several limitations. Its structured memory design trades visual fidelity for searchability: converting visual evidence into textual fields makes it compatible with event-level retrieval, but may lose fine-grained pixel details such as small objects, colors, layouts, and subtle visual states. Future work may explore coarse-to-fine multimodal memory, where structured fields support retrieval while finer visual evidence is preserved for verification. The align-then-retrieve paradigm is also not fully complete. Although multimodal evidence is aligned under event anchors before retrieval, the final answer stage still relies on selected keyframes for visually detailed reasoning. This leaves part of cross-modal alignment to inference time and may introduce modality bias. EM²Mem also depends on upstream MLLMs and visual tools, so errors in captioning, object extraction, or metadata normalization may propagate into event cells and graphs. In addition, the framework shifts computation from inference to memory construction, making it more suitable for videos processed once and queried repeatedly than for real-time scenarios.

EM²Mem 存在若干局限。 其结构化记忆设计以视觉保真度换取可检索性：将视觉证据转换为文本字段，使其适用于事件级检索，但可能丢失小物体、颜色、布局和细微视觉状态等精细像素信息。 未来工作可探索由粗到细的多模态记忆，由结构化字段支持检索，同时保留更精细的视觉证据用于验证。 先对齐、后检索的范式也尚未完全实现。 虽然多模态证据已在检索前通过事件锚点对齐，但最终作答阶段仍依赖选定的关键帧，才能进行涉及视觉细节的推理。 这使部分跨模态对齐仍留在推理阶段，并可能引入模态偏差。 EM²Mem 还依赖上游 MLLM 和视觉工具，因此描述生成、对象提取或元数据规范化中的错误，可能传播到事件单元和图中。 此外，该框架将计算从推理阶段转移到记忆构建阶段，因此更适合一次处理、多次查询的视频，而非实时场景。

Ethical Statement

This work studies long-video question answering with multimodal memory, including evaluations on egocentric daily-life benchmarks. Such videos may contain privacy-sensitive information, including personal routines, indoor environments, object usage, social interactions, and potentially identifying visual or textual cues. We do not collect new human-subject data, and all experiments are conducted on publicly released benchmark datasets under their intended research settings. We do not attempt to identify individuals, infer protected attributes, or disclose raw video content or personally identifiable information; all results are reported in aggregate. Since EM²Mem organizes events, entities, routines, and relations into structured memory, misuse on unconstrained personal videos could increase risks of privacy leakage, behavioral profiling, or surveillance-like applications. Therefore, practical deployment of such memory systems should require informed consent, access control, data minimization, memory deletion or expiration mechanisms, and safeguards against using inferred semantic facts for high-stakes decisions. We also note that egocentric video benchmarks may reflect demographic and environmental biases, so performance should not be assumed to generalize uniformly across populations or deployment contexts.

本研究探讨使用多模态记忆进行长视频问答，包括在第一人称日常生活基准上的评测。 这类视频可能包含涉及隐私的敏感信息，包括个人日常活动、室内环境、物品使用、社交互动，以及可能用于识别身份的视觉或文本线索。 我们未收集新的人类受试者数据，所有实验均在公开发布的基准数据集上，按照其预期研究用途开展。 我们未尝试识别个人、推断受保护属性，或披露原始视频内容及个人可识别信息；所有结果均以汇总形式报告。 由于 EM²Mem 将事件、实体、日常规律和关系组织为结构化记忆，在未受约束的个人视频上滥用该系统，可能增加隐私泄露、行为画像或类似监控应用的风险。 因此，这类记忆系统的实际部署应要求知情同意、访问控制、数据最小化、记忆删除或到期机制，并设置防护措施，避免将推断出的语义事实用于高风险决策。 我们也注意到，第一人称视频基准可能反映人口特征和环境方面的偏差，因此不应假设其性能能够在不同人群或部署情境中一致泛化。