A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
MemoryAgentICML 2024CCF-AGoogle DeepMindhttps://github.com/read-agent/read-agent.github.io/blob/main/assets/read_agent_demo.ipynb
具备超长上下文要点记忆的人类启发式阅读智能体
Abstract
Current Large Language Models (LLMs) are not only limited to some maximum context length, but also are not able to robustly consume long inputs. To address these limitations, we propose ReadAgent, an LLM agent system that increases effective context length up to
当前的大语言模型（LLM）不仅受限于某个最大上下文长度，也无法稳健地处理长输入。 为解决这些限制，我们提出 ReadAgent，这是一种在实验中将有效上下文长度提升至多
1. Introduction
Transformer-based Large Language Models (LLMs) are highly capable of language understanding, but the amount of text that LLMs are able to read at one time is constrained. Not only is there an explicit context length limitation, but it has also been found that performance of LLMs tends to decline with increasingly long inputs even when they don't actually exceed the explicit context window. In contrast, humans can read, understand, and reason over very long texts, such as a series of interrelated books.
基于 Transformer 的大语言模型（LLM）具有很强的语言理解能力，但 LLM 一次能够读取的文本量受到限制。 它们不仅存在明确的上下文长度限制，而且研究还发现，即使长输入实际上没有超过明确的上下文窗口，LLM 的性能也往往会随着输入变长而下降。 相比之下，人类能够阅读、理解并推理非常长的文本，例如一系列相互关联的书籍。
We posit that an underlying reason for this gap is inherent in the differences in reading approaches. Typically, we use LLMs to consume the exact given content word-by-word and the process is relatively passive. On the other hand, humans read and reason over long text differently. First, the exact information tends to be forgotten quickly, whereas the fuzzier gist information, i.e. the substance irrespective of exact words, from past readings lasts much longer. Second, human reading is an interactive process. When we need to remind ourselves of relevant details in order to complete a task, such as answering a question, we look them up in the original text.
我们认为，这一差距的一个根本原因在于阅读方式的差异。 通常，我们让 LLM 逐字处理给定的确切内容，而这个过程相对被动。 另一方面，人类阅读和推理长文本的方式有所不同。 首先，确切信息往往很快被遗忘，而过去阅读中更模糊的要点信息，也就是不依赖确切措辞的内容实质，却会保留更久。 其次，人类阅读是一个交互过程。 当我们为了完成某项任务，例如回答问题，而需要回想相关细节时，我们会回到原文中查找这些细节。
We think that using the fuzzy gist memory to capture global context and attending to local details together enables humans to reason over very long context efficiently, in terms of how much information to process at once, and is also important for comprehension. For example, if we were to infer the intention of a fictional character's specific action described on a page in a novel, besides focusing on the surrounding pages, we likely also need to understand the overall story and the character's personality from reading the whole book (see the appendix for more analysis).
我们认为，使用模糊的要点记忆捕获全局上下文，同时关注局部细节，能够让人类从一次需要处理的信息量角度高效推理超长上下文，并且对理解也很重要。 例如，如果我们要推断小说某一页所描述的虚构人物某项具体行动的意图，除了关注相邻页面，我们很可能还需要通过阅读整本书来理解完整故事和该人物的性格（更多分析见附录）。
Motivated by these observations, we propose ReadAgent, an LLM agent system that handles long content inspired by the human approach. ReadAgent is simple to implement and can be built entirely by prompting a previously-trained LLM. As illustrated in Figure 1, it takes three primary steps: (1) episode pagination, where we prompt the LLM to decide where to pause in reading contiguous text; the content between pause points becomes an episode, which we refer to as pages in this work; (2) memory gisting, where we prompt the LLM to compress each page into a shorter gist and associate the gist with a corresponding context (e.g. which page the gist was from) -- this gives the episodic gist memory; (3) interactive look-up, where the LLM looks at the given task and the complete set of gists in-context, makes decision on what page(s) to look up, combines the gists with these raw pages, and solves the task.
受这些观察启发，我们提出 ReadAgent，这是一种借鉴人类方式处理长内容的 LLM 智能体系统。 ReadAgent 实现简单，只需提示一个已经训练好的 LLM 即可完整构建。 如图1所示，它包含三个主要步骤： （1）情节分页：我们提示 LLM 决定在连续文本的什么位置暂停阅读；两个暂停点之间的内容构成一个情节，本文将其称为页面； （2）记忆提要：我们提示 LLM 把每个页面压缩成更短的要点，并将要点与对应上下文关联起来（例如该要点来自哪个页面），由此得到情景式要点记忆； （3）交互式查阅：LLM 在上下文中查看给定任务和完整要点集合，决定要查阅哪些页面，把要点与这些原始页面合并，然后解决任务。
We evaluate ReadAgent by comparing against using only the gist memory without interactive look-up, using full text for datasets that can fit in the context window, and using retrieval methods to look up pages. ReadAgent outperforms all baselines across three challenging long-document comprehension tasks -- QuALITY, NarrativeQA and QMSum -- while increasing the effective context length significantly compared to the original LLM. On NarrativeQA Gutenberg test set, whose average length is 71k words and whose maximum is 343k words, ReadAgent improves the LLM rating (Section 4.1) by 12.97% and ROUGE-L by 31.98% over the best retrieval baseline and increases the effective context length by
我们通过以下方式评估 ReadAgent：与不进行交互式查阅、仅使用要点记忆的方法比较；在数据能够装入上下文窗口时与使用全文的方法比较；以及与使用检索方法查阅页面的方法比较。 在 QuALITY、NarrativeQA 和 QMSum 三项具有挑战性的长文档理解任务上，ReadAgent 均优于所有基线，同时相比原始 LLM 显著增加了有效上下文长度。 在 NarrativeQA Gutenberg 测试集上，文本平均长度为 7.1 万词、最大长度为 34.3 万词；与最佳检索基线相比，ReadAgent 将 LLM 评分（第 4.1 节）提高了 12.97%，将 ROUGE-L 提高了 31.98%，并把有效上下文长度提升了约
Finally, in the appendix, we adapt ReadAgent to web navigation, which is a fundamentally very-long context agent setting. We find that ReadAgent is simple to adapt to this setting and shows promising performance.
最后，我们在附录中把 ReadAgent 适配到网页导航，这从根本上属于一种超长上下文智能体场景。 我们发现，ReadAgent 很容易适配到该场景，并展现出很有前景的性能。
Our primary contributions are:
- ReadAgent, our human-inspired LLM agent that generates gist memories and looks up information as needed for solving tasks on long contexts (Section 3).
- Demonstration of significant performance advantages and scalability through a comprehensive experimental evaluation on challenging long-context benchmarks, comparisons against popular baselines, and analysis (Section 4).
我们的主要贡献如下：
- ReadAgent：我们提出的人类启发式 LLM 智能体，它生成要点记忆，并在解决长上下文任务时按需查阅信息（第 3 节）。
- 通过在具有挑战性的长上下文基准上开展全面实验评估、与常用基线比较并进行分析，证明了显著的性能优势与可扩展性（第 4 节）。
2. Related Work
Long-Context LLMs. The most direct way to improve LLM long-context performance is to train or fine-tune LLMs with longer context windows. Another approach is to explore new architectures or efficient implementations of the Transformer attention layers to reduce the need of long-context fine-tuning. However, LLM performance tends to decline with increasingly long inputs even when they don't exceed the specified context length. LLM performance is also shown to be sensitive to distracting information in the context. Thus, the effective context length could be shorter than the explicit limit. Our approach is complimentary to these approaches, scaling the effective context length of the underlying model while reducing the amount of distracting information in context, and requiring neither architectural changes nor training.
长上下文 LLM。 提升 LLM 长上下文性能最直接的方法，是使用更长的上下文窗口训练或微调 LLM。 另一种方法是探索新架构或 Transformer 注意力层的高效实现，以减少对长上下文微调的需求。 然而，即使输入没有超过指定的上下文长度，LLM 性能也往往会随着输入变长而下降。 研究还表明，LLM 性能对上下文中的干扰信息很敏感。 因此，有效上下文长度可能短于明确规定的上限。 我们的方法与这些方法互补：它在减少上下文干扰信息的同时扩展底层模型的有效上下文长度，并且既不需要修改架构，也不需要训练。
Retrieval. Retrieval Augmented Generation (RAG) techniques allow an LLM to query task-relevant information from a large database of documents or document pieces. Our work implements a form of retrieval by reasoning over a contextualized gist memory, all with zero-shot LLM prompting. This rethinking of retrieval directly leverages the strength and flexibility of LLM language understanding to reason about which documents to retrieve. Our approach is well-suited to densely-correlated long-document pieces, such as a series of books or a conversation history, but the database cannot scale arbitrarily, since the size of the gist memory is limited by the LLM's context length, and the gist memory's length correlates with the size of the database. In contrast, conventional retrieval approaches can handle larger databsases than our approach. In this work, we compare against retrieval systems that use exactly the same set of documents as our approach.
检索。 检索增强生成（RAG）技术允许 LLM 从大型文档或文档片段数据库中查询与任务相关的信息。 我们的工作通过对带有上下文的要点记忆进行推理来实现一种检索形式，整个过程仅使用零样本 LLM 提示。 这种对检索的重新思考直接利用了 LLM 语言理解的能力与灵活性，以推理应检索哪些文档。 我们的方法很适合相关性密集的长文档片段，例如一系列书籍或对话历史；但数据库不能无限扩展，因为要点记忆的大小受 LLM 上下文长度限制，而且要点记忆长度与数据库规模相关。 相比之下，传统检索方法能够处理比我们的方法更大的数据库。 在本文中，我们与使用完全相同文档集合的检索系统进行比较。
LLM Agents for Long Texts. LLMs can be used as agents to interactively handle very long texts. WebGPT and WebShop learn browsing actions to search for the requested answer on the internet, despite not being designed to understand long documents. The PEARL system proposes action plans for better long-document comprehension through iterative prompting; Yuan et al. explicitly learns RL agents for similar purposes. Self-note amortizes reasoning steps and interleaves intermediate notes with the original documents to improve reasoning. Yang et al. generates long outputs through iterative reasoning. However, these methods cannot address long input texts that exceed the LLM's context length. Similar to this work, MemWalker also reads long documents interactively through iterative prompting. It traverses a tree of different levels of summaries to search for task-related information. However, the hierarchical summary structure makes it difficult to reason over related but distant information at the same granularity (see the appendix for more discussion).
面向长文本的 LLM 智能体。 LLM 可以作为智能体，以交互方式处理非常长的文本。 尽管 WebGPT 和 WebShop 并非为理解长文档而设计，它们仍会学习浏览动作，以便在互联网上搜索所需答案。 PEARL 系统通过迭代提示提出行动计划，以改善长文档理解；Yuan 等人则为类似目标显式训练强化学习智能体。 Self-note 对推理步骤进行摊销，并在原始文档之间插入中间笔记以改善推理。 Yang 等人通过迭代推理生成长输出。 然而，这些方法无法处理超过 LLM 上下文长度的长输入文本。 与本工作类似，MemWalker 也通过迭代提示以交互方式阅读长文档。 它遍历由不同层级摘要构成的树，以搜索与任务相关的信息。 然而，层级摘要结构使系统难以在相同粒度上对彼此相关但相距遥远的信息进行推理（更多讨论见附录）。
3. ReadAgent
Figure 1 shows an overview of ReadAgent, which we describe in detail below. Note that the prompts presented in this section are examples, which may need to change according to the target task. We release the prompts for each task on read-agent.github.io. Please also refer to the appendix for the prompt design details.
图 1 展示了 ReadAgent 的概览，下面将详细介绍。 请注意，本节给出的提示只是示例，可能需要根据目标任务进行调整。 我们在 read-agent.github.io 发布了各项任务所用的提示。 提示设计细节另见附录。
3.1 Gist Memory
A gist memory is an ordered collection of short gists of chunks of text from the original long context. Building a gist memory has two steps: pagination and memory gisting, described in turn below.
要点记忆是由原始长上下文中各文本块的简短要点按顺序组成的集合。 构建要点记忆包含两个步骤：分页与记忆提炼，下面依次介绍。
Episode Pagination. When ReadAgent reads through a long text, it makes decisions on what content to store together in a memory episode by choosing where to pause reading. At each step, we provide the LLM some text that begins from the previous pause point and ends when it reaches a max_words limit. We prompt the LLM to choose which point between paragraphs would be a natural point to pause, and then treat the content between the previous and current pause points as an episode, which we also refer as a page. This is episode pagination, which we implement with the following prompt.
情节分页。 当 ReadAgent 阅读一段长文本时，它通过选择暂停阅读的位置，决定哪些内容应共同存入一个记忆情节。 在每一步中，我们向 LLM 提供一段从上一个暂停点开始、到达到 max_words 限制时结束的文本。 我们提示 LLM 在段落之间选择一个自然的暂停点，然后把上一个暂停点与当前暂停点之间的内容视为一个情节，也称为一页。 这就是情节分页，我们使用下面的提示来实现它。
As shown in the prompt, possible pause points are inserted between paragraphs as numbered tags (e.g. min_words threshold to make sure that each page has at least min_words.
如提示所示，候选暂停点以编号标签（例如 min_words 阈值后，我们才开始插入这些编号标签，以确保每页至少包含 min_words 个词。
You are given a passage that is taken from a larger text (article, book, ...) and some numbered labels between the paragraphs in the passage.
Numbered labels are in angle brackets. For example, if the label number is 19, it shows as <19> in text.
Please choose a label where it is natural to break reading.
The label can be a scene transition, the end of a dialogue, the end of an argument, a narrative transition, etc.
Please answer with the break point label and explain.
For example, if <57> is a good point to break, answer with Break point: <57>\n Because ...
Passage:
{...}
{PARAGRAPH 5 TEXT}
<5>
{PARAGRAPH 6 TEXT}
<6>
{PARAGRAPH 7 TEXT}
{...}你会得到一个取自更长文本（文章、书籍等）的段落，以及插在该段落各自然段之间的一些编号标签。
编号标签放在尖括号中。例如，编号为 19 的标签在文本中显示为 <19>。
请选择一个适合自然暂停阅读的标签。
该标签可以位于场景转换、对话结束、论证结束、叙事转换等位置。
请回答断点标签并说明原因。
例如，如果 <57> 是合适的断点，请回答 Break point: <57>\n Because ...
文本：
{...}
{第 5 段文本}
<5>
{第 6 段文本}
<6>
{第 7 段文本}
{...}Memory Gisting. For each page, we prompt the LLM to shorten the exact content into a gist, or summary, as follows.
记忆提炼。 对于每一页，我们使用下面的提示，让 LLM 把确切内容缩短为一个要点或摘要。
Please shorten the following passage.
Just give me a shortened version. DO NOT explain your reason.
Passage:
{PAGE TEXT}请缩短下面的文本。
只给出缩短后的版本。不要解释原因。
文本：
{页面文本}We subsequently prepend a page tag to each gist (e.g. “
随后，我们在每个要点前加上页码标签（例如“
The original page size is a key factor for how compressed the gist is. Let's say the smallest unit of text that we consider is a paragraph. Intuitively, a paragraph likely has some amount of mutual information with its neighbors. Thus, the larger chunk of text we group together, the more duplicated information we can remove. Empirically, compressing larger chunks of text with LLMs also tends to remove more details, which could affect performance. We control the page size by changing min_words and max_words in pagination. This trade-off is studied in Section 4.4.
原始页面大小是决定要点压缩程度的关键因素。 假设我们考虑的最小文本单位是一个段落。 直观上，一个段落很可能与相邻段落具有一定的互信息。 因此，我们组合的文本块越大，就能删除越多重复信息。 从经验上看，使用 LLM 压缩更大的文本块也往往会删去更多细节，这可能影响性能。 我们通过改变分页中的 min_words 和 max_words 来控制页面大小。 第 4.4 节研究了这一权衡。
3.2 Interactive Look-Up and Response
For a given task about a long document, we want ReadAgent to take actions to look up relevant details in the original text in addition to using its gist memory. As the gist memories are contextualized with page numbers, we simply prompt the LLM to answer which page(s) it would like to look up and read again given the specific task. In the following we discuss two look-up strategies: looking up all pages at once in parallel (ReadAgent-P) and sequentially looking up one page at a time (ReadAgent-S).
对于给定的长文档任务，除了使用要点记忆外，我们还希望 ReadAgent 采取行动，在原文中查找相关细节。 由于要点记忆带有页码上下文，我们只需提示 LLM 根据具体任务回答它希望查阅并重新阅读哪些页面。 下面讨论两种查阅策略：一次并行查阅所有页面（ReadAgent-P），以及每次顺序查阅一页（ReadAgent-S）。
ReadAgent-P. As in the following example prompt for question-answering, typically we give it a maximum number of pages that it can look up but also instruct it to use as few pages as possible to avoid unnecessary computational overhead and distracting information. The following prompt shows parallel look-up, where the model requests multiple pages in response to a single prompt.
ReadAgent-P。 如下面的问答提示示例所示，我们通常规定它最多可以查阅多少页，同时也要求它尽量少用页面，以避免不必要的计算开销和干扰信息。 下面的提示展示了并行查阅，模型会响应一次提示而请求多个页面。
The following text is what you remember from reading an article and a multiple choice question related to it.
You may read 1 to 5 page(s) of the article again to refresh your memory to prepare yourself for the question.
Please respond with which page(s) you would like to read.
For example, if you only need to read Page 8, respond with I want to look up Page [8] to ...; if you would like to read Page 7 and 12, respond with I want to look up Page [7, 12] to ...; if you would like to read Page 2, 3, 7, 15 and 18, respond with I want to look up Page [2, 3, 7, 15, 18] to ....
DO NOT select more pages if you don't need to.
You don't need to answer the question yet.
Text:
{GIST MEMORY}Question:
{QUESTION}下面的文本是你阅读一篇文章后记住的内容，随后是一道与文章相关的多项选择题。
你可以重新阅读文章的 1 至 5 页，以唤起记忆并为回答问题做准备。
请回答你希望阅读哪些页面。
例如，只需阅读第 8 页时，回答 I want to look up Page [8] to ...；希望阅读第 7、12 页时，回答 I want to look up Page [7, 12] to ...；希望阅读第 2、3、7、15、18 页时，回答 I want to look up Page [2, 3, 7, 15, 18] to ...。
如果不需要，请不要选择更多页面。
你暂时不需要回答问题。
文本：
{要点记忆}问题：
{问题}The selected raw pages replace the gist(s) at the corresponding positions in memory, preserving the overall narrative flow. Then we prompt the LLM again with the task and the updated memory and ask it to solve the task (see example prompts in the appendix).
选中的原始页面会替换记忆中对应位置的要点，从而保留整体叙事流。 然后，我们再次向 LLM 提供任务和更新后的记忆，并要求它解决任务（提示示例见附录）。
ReadAgent-S. We also study the sequential look-up strategy, where the model requests one page at a time, up to some maximum number of pages. In sequential look-up, the model gets to see the previously expanded pages before deciding which page to expand. This gives the model access to more information than parallel look-up, so we might expect it to perform better in some situations. However, the larger number of interactions with the model increases the computational cost, so sequential look-up should only be used on tasks where it provides clear benefits.
ReadAgent-S。 我们还研究了顺序查阅策略：模型每次请求一页，直到达到某个最大页数。 在顺序查阅中，模型会在决定下一页展开哪一页之前看到此前已经展开的页面。 因此，模型能比并行查阅获得更多信息，所以在某些情况下可能表现更好。 然而，与模型进行更多次交互会提高计算成本，因此顺序查阅只应在能够带来明确收益的任务中使用。
The following text is what you remember from reading a meeting transcript, followed by a question about the transcript.
You may read multiple pages of the transcript again to refresh your memory and prepare to answer the question.
Each page that you re-read can significantly improve your chance of answering the question correctly.
Please specify a SINGLE page you would like to read again or say STOP.
To read a page again, respond with Page $PAGE_NUM, replacing $PAGE_NUM with the target page number.
You can only specify a SINGLE page in your response at this time.
To stop, simply say STOP. DO NOT answer the question in your response.
Text:
{GISTS WITH IN-LINE EXPANDED PAGES}Pages re-read already (DO NOT ask to read them again):
{LIST OF PAGE NUMBERS ALREADY READ}Question:
{QUESTION}Specify a SINGLE page to read again, or say STOP:
下面的文本是你阅读一份会议记录后记住的内容，随后是一个关于该记录的问题。
你可以重新阅读记录中的多个页面，以唤起记忆并准备回答问题。
每重新阅读一页，都能显著提高你正确回答问题的概率。
请指定你想重新阅读的一个页面，或者回答 STOP。
要重新阅读某页，请回答 Page $PAGE_NUM，并用目标页码替换 $PAGE_NUM。
此时你的回答中只能指定一个页面。
若要停止，只需回答 STOP。不要在回答中作答问题。
文本：
{含行内展开页面的要点}已经重读的页面（不要再次请求）：
{已经阅读的页码列表}问题：
{问题}指定一个要重新阅读的页面，或回答 STOP：
3.3 Computational Trade-offs and Scalability
Episode pagination, memory gisting and interactive look-ups require iterative inference. As we show in the following, the additional cost is bounded linearly by a small factor, making our approach scale well with input length.
情节分页、记忆提炼和交互式查阅都需要迭代推理。 如下所示，额外成本由一个较小的线性因子限定，因此我们的方法能够随输入长度良好扩展。
Pagination. In theory, an LLM could read a document and directly provide the pagination in a single pass, so the minimum number of words the LLM must process is the length of the document. Our pagination algorithm splits the document into chunks of at most max_words, and then guarantees that at least min_words are consumed at each step. Thus, the ratio
分页。 理论上，LLM 可以阅读一篇文档并在一次处理中直接给出分页结果，因此 LLM 必须处理的最少词数就是文档长度。 我们的分页算法把文档划分为至多包含 max_words 个词的块，并保证每一步至少处理 min_words 个词。 因此，比值
Gisting: Memory gisting is one additional pass of the raw input words, since each page is gisted independently. Look-ups: Parallel look-ups are conditioned on gists instead of the full text, and thus will be much shorter than one pass of the raw input words. Each step of a sequential look-up is similar to parallel look-ups and the overall cost is capped with the maximum number of look-ups allowed. Response: Finally, answering is also similar to parallel look-ups. There is additional overhead from the prompt templates, of course.
提炼： 由于每页都独立提炼，记忆提炼相当于额外遍历一次原始输入词。 查阅： 并行查阅以要点而非全文为条件，因此会比遍历一次原始输入词短得多。 顺序查阅的每一步与并行查阅相似，而总成本受允许查阅的最大次数限制。 回答： 最后的作答也与并行查阅相似。 当然，提示模板还会带来额外开销。
On the other hand, as generating gists is an one-time effort while the look-up and response steps operate mostly on gists that are much shorter than the original text, the one-time effort can be amortized when the same context is reused for multiple tasks. Thus, in such settings, ReadAgent can reduce the overall number of tokens to process. In particular, directly answering from the original QuALITY dev set (230 articles and 2086 questions) is 8,708,434 words consumed by the LLM, whereas using ReadAgent with 1-page lookup is 6,499,856 words (25.4% saving), up-to-2-page lookup is 6,933,357 words (20.4% saving), and up-to-5-page lookup is 7,503,084 words (13.8% saving). We can expect the savings to be more significant with higher compression rate and more downstream tasks.
另一方面，生成要点只需一次，而查阅和回答步骤主要处理远短于原文的要点；当同一上下文被多个任务复用时，这项一次性工作可以被摊销。 因此，在这类设置中，ReadAgent 可以减少需要处理的 token 总数。 具体而言，直接回答原始 QuALITY 开发集（230 篇文章、2086 个问题）会让 LLM 处理 8,708,434 个词；使用 ReadAgent 查阅 1 页时处理 6,499,856 个词（节省 25.4%），最多查阅 2 页时处理 6,933,357 个词（节省 20.4%），最多查阅 5 页时处理 7,503,084 个词（节省 13.8%）。 压缩率越高、下游任务越多，预期节省幅度就越显著。
3.4 ReadAgent Variants
In the appendix, we discuss variants of ReadAgent that can be useful in different problem settings, including when the target task is known prior to reading the long document. We also describe adapting ReadAgent to work in the web navigation setting.
附录讨论了适用于不同问题设置的 ReadAgent 变体，包括在阅读长文档之前已经知道目标任务的情况。 我们还介绍了如何调整 ReadAgent，使其适用于网页导航设置。
4. Experiments
We evaluate ReadAgent's long-document reading comprehension ability on three long-context question-answering challenges: QuALITY, NarrativeQA and QMSum. Although ReadAgent does not require any model training, we develop the proposed method on the training sets and test on the validation, test and/or development sets to avoid any risk of overfitting system hyperparameters.
我们在三个长上下文问答挑战上评估 ReadAgent 的长文档阅读理解能力：QuALITY、NarrativeQA 和 QMSum。 虽然 ReadAgent 不需要任何模型训练，但我们在训练集上开发所提出的方法，并在验证集、测试集和/或开发集上测试，以避免系统超参数过拟合的风险。
In this work, we primarily use the instruction-tuned PaLM 2-L for our experiments and evaluation. The context length of PaLM 2-L is 8K tokens. Details of the model can be found in the PaLM 2 technical report. Additionally, we provide GPT-3.5 results and experimental results on the web navigation setting in the appendix.
在本文中，我们主要使用经过指令微调的 PaLM 2-L 进行实验与评估。 PaLM 2-L 的上下文长度为 8K token。 模型细节见 PaLM 2 技术报告。 此外，附录还给出了 GPT-3.5 结果和网页导航设置下的实验结果。
One important performance measure of the techniques considered here is the compression rate (CR). As we want to measure the longest LLM context length that ReadAgent requires versus full-context length, we define this at the final response query as follows.
本文所考察技术的一项重要性能指标是压缩率（CR）。 为了比较 ReadAgent 所需的最长 LLM 上下文长度与完整上下文长度，我们在最终回答查询处将其定义如下。
The in-context text (gists and retrieved pages) length is the longest among all inference steps.
上下文内文本（要点和检索到的页面）的长度取所有推理步骤中的最大值。
4.1 LLM Raters
NarrativeQA and QMSum both have one or more free-form reference responses. They are typically evaluated using syntactic matching metrics such as ROUGE F-Measure. We additionally evaluate these datasets using an automatic LLM Rater as an alternative to human evaluation similar to prior work.
NarrativeQA 和 QMSum 都有一个或多个自由形式的参考回答。 它们通常使用 ROUGE F 值等句法匹配指标进行评估。 与先前工作类似，我们还使用自动 LLM 评审器评估这些数据集，以替代人工评估。
In our implementation, we prompt the LLM to look at the question or instruction and compare the model's answer to the reference answer. The “Strict LLM Rater Prompt” shown below is for judging whether there is an exact match, and the “Permissive LLM Rater Prompt” is for judging whether there is an exact match or a partial match. We apply both prompts to all model responses. If either rater decides there is an exact match, we count it as an exact match. If the strict rater is negative but the permissive rater detects a partial match, we count it as a partial match. Otherwise, it's not a match. In the case that there are multiple reference answers, the response is compared against each reference answer in turn, and the highest rating is returned.
在我们的实现中，我们提示 LLM 查看问题或指令，并把模型回答与参考回答进行比较。 下面的“严格 LLM 评审提示”用于判断是否完全匹配，“宽松 LLM 评审提示”用于判断是否完全匹配或部分匹配。 我们把两个提示都应用于所有模型回答。 如果任一评审器判定完全匹配，我们就将其计为完全匹配。 如果严格评审器给出否定结果，而宽松评审器检测到部分匹配，我们就将其计为部分匹配。 否则，视为不匹配。 当存在多个参考回答时，模型回答会依次与每个参考回答比较，并返回最高评分。
After reading some text, John was given the following question about the text:
{QUESTION TEXT}John's answer to the question was:
{MODEL RESPONSE TEXT}The ground truth answer was:
{REFERENCE RESPONSE TEXT}Does John's answer agree with the ground truth answer? Please answer YES or NO.
John 阅读一段文本后，得到了下面这个关于文本的问题：
{问题文本}John 对问题的回答是：
{模型回答文本}标准答案是：
{参考回答文本}John 的回答是否与标准答案一致？请回答 YES 或 NO。
After reading some text, John was given the following question about the text:
{QUESTION TEXT}John's answer to the question was:
{MODEL RESPONSE TEXT}The ground truth answer was:
{REFERENCE RESPONSE TEXT}Does John's answer agree with the ground truth answer? Please answer “Yes”, “Yes, partially”, or “No”.
If John's response has any overlap with the ground truth answer, answer “Yes, partially”.
If John's response contains the ground truth answer, answer “Yes”.
If John's response is more specific than the ground truth answer, answer “Yes”.
John 阅读一段文本后，得到了下面这个关于文本的问题：
{问题文本}John 对问题的回答是：
{模型回答文本}标准答案是：
{参考回答文本}John 的回答是否与标准答案一致？请回答“Yes”“Yes, partially”或“No”。
如果 John 的回答与标准答案有任何重合，请回答“Yes, partially”。
如果 John 的回答包含标准答案，请回答“Yes”。
如果 John 的回答比标准答案更具体，请回答“Yes”。
Based on these raters, we define two different scores: LLM-Rating-1 (LR-1) is a strict evaluation score, where we count the percentage of exact matches over all examples; LLM-Rating-2 (LR-2) is permissive, where we count the percentage of exact and partial matches.
基于这些评审器，我们定义两个不同分数：LLM-Rating-1（LR-1）是严格评估分数，统计所有样本中完全匹配的比例；LLM-Rating-2（LR-2）是宽松评估分数，统计完全匹配和部分匹配的比例。
4.2 Baseline Methods
Retrieval-Augmented Generation (RAG). As discussed in Section 2, RAG is a popular approach to extend access to a large amount of text beyond what can fit in the LLM context window. In this paper we compare ReadAgent to RAG baselines using conventional retrieval methods to find relevant “pages” in a long text, where we reuse the pages generated by ReadAgent. We consider two relevance methods: Okapi BM25 and neural retrieval based on the Gemini API embedding model (models/embedding-001). The neural retrieval relevance score is defined as the dot product between the question embedding vector and each page (or gist memory embedding vector in the case of NarrativeQA, see Section 4.3.2). For reading comprehension tasks, the pages are ranked by relevance to each question, and we prompt the LLM to look at the top-
检索增强生成（RAG）。 如第 2 节所述，RAG 是一种常用方法，用于访问超出 LLM 上下文窗口容量的大量文本。 本文使用传统检索方法从长文本中寻找相关“页面”，并复用 ReadAgent 生成的页面，把 ReadAgent 与 RAG 基线进行比较。 我们考虑两种相关性方法：Okapi BM25，以及基于 Gemini API 嵌入模型（models/embedding-001）的神经检索。 神经检索的相关性分数定义为问题嵌入向量与每个页面嵌入向量的点积；在 NarrativeQA 中则使用要点记忆嵌入向量，见第 4.3.2 节。 对于阅读理解任务，页面按照与各问题的相关性排序，我们提示 LLM 把前
In most retrieval settings, the database of documents is quite large, which makes the retrieval task more challenging. In our setting, ReadAgent and retrieval methods all use a per-document database, rather than per-dataset. For example, in QuALITY, there are hundreds of articles, each with multiple questions. The database for retrieval in each question is only the extracted pages from the corresponding article (typically less than 20 pages), rather than the thousands of pages from the entire dataset.
在大多数检索设置中，文档数据库相当庞大，因此检索任务更具挑战性。 在我们的设置中，ReadAgent 和检索方法都使用按文档建立的数据库，而不是按数据集建立数据库。 例如，QuALITY 包含数百篇文章，每篇文章有多个问题。 每个问题的检索数据库只包含从相应文章中提取的页面，通常少于 20 页，而不是整个数据集中的数千页。
Full or Truncated Text Content. The maximum length of QuALITY dev articles is
完整或截断的文本内容。 QuALITY 开发集文章的最大长度约为
Gist Memory. We can also attempt to solve the given task by reasoning directly over the gist memory. Doing so helps us understand not only the importance of interactive look-up but also how using the LLM-compressed information alone compares to the full content and retrieval baselines.
要点记忆。 我们还可以尝试直接对要点记忆进行推理来解决给定任务。 这样不仅有助于理解交互式查阅的重要性，也有助于了解单独使用 LLM 压缩信息与完整内容和检索基线相比表现如何。
4.3 Long-Context Reading Comprehension
4.3.1 QuALITY
QuALITY is a four-way multiple choice question answering challenge with text data from several different sources. QuALITY is evaluated using accuracy, with 25% corresponding to chance performance.
QuALITY 是一个四选一问答挑战，其文本数据来自若干不同来源。 QuALITY 使用准确率进行评估，25% 对应随机猜测水平。
The dev set has an average length of 4,122 words and a maximum of 5,967. The gist memory has an average length of 650 words and a maximum of 1,264. Figure 2 shows the word statistics for the original text and the gists. The compression rate of the gists is 85.53%. See the appendix for QuALITY pagination hyperparameters.
开发集的平均长度为 4,122 个词，最大长度为 5,967 个词。 要点记忆的平均长度为 650 个词，最大长度为 1,264 个词。 图 2 展示了原始文本与要点的词数统计。 要点的压缩率为 85.53%。 QuALITY 的分页超参数见附录。
Table 1 shows the experimental results on QuALITY. The performance of ReadAgent increases as we increase the maximum number of pages allowed for look-up. ReadAgent-P (Look up 1-6 pages) achieves 86.91% and ReadAgent-S (Look up 1-6 pages) achieves 87.17% in accuracy. Notably, starting from ReadAgent (Look up 1-2 pages), it outperforms all baselines methods including using the full original text, which could have been an upper bound on the performance -- every other method reduces the amount of text the LLM considers before generating its response. However, this is not a surprising result.
表 1 展示了 QuALITY 上的实验结果。 随着允许查阅的最大页数增加，ReadAgent 的性能也随之提高。 ReadAgent-P（查阅 1 至 6 页）的准确率达到 86.91%，ReadAgent-S（查阅 1 至 6 页）达到 87.17%。 值得注意的是，从 ReadAgent（查阅 1 至 2 页）开始，它就优于包括使用完整原文在内的所有基线；完整原文本可视为性能上界，因为其他所有方法都会在生成回答前减少 LLM 所考虑的文本量。 不过，这个结果并不令人意外。
Prior work shows that current LLMs are not able to effectively use the full long context window, potentially due to training data sparsity, and distracting information can also reduce performance. The corresponding compression rate of ReadAgent (Look up 1-2 pages) is 72.17%, meaning that
先前工作表明，当前 LLM 无法有效利用完整的长上下文窗口，这可能源于训练数据稀疏；干扰信息也会降低性能。 ReadAgent（查阅 1 至 2 页）对应的压缩率为 72.17%，这意味着提炼要点后，上下文窗口可以容纳约
| Method | CR (# LU) | Accuracy |
|---|---|---|
| BM25 Retrieval | ||
| Top-1 | 89.27% (1) | 70.34% ± 0.06 |
| Top-2 | 78.96% (2) | 79.05% ± 0.05 |
| Top-3 | 68.50% (3) | 82.65% ± 0.05 |
| Top-4 | 58.57% (4) | 84.42% ± 0.13 |
| Neural Retrieval with Gemini API | ||
| Top-1 | 89.91% (1) | 71.32% ± 0.19 |
| Top-2 | 80.08% (2) | 79.02% ± 0.10 |
| Top-3 | 70.28% (3) | 83.41% ± 0.10 |
| Top-4 | 60.68% (4) | 84.88% ± 0.03 |
| Full Raw Content | 0% | 85.83% ± 0.19 |
| GistMem | 85.53% | 77.52% ± 0.13 |
| ReadAgent-P | ||
| Look up 1 pg | 76.00% (1.0) | 84.13% ± 0.10 |
| Look up 1-2 pgs | 72.17% (1.6) | 86.16% ± 0.12 |
| Look up 1-3 pgs | 69.36% (2.0) | 86.59% ± 0.10 |
| Look up 1-4 pgs | 67.73% (2.2) | 86.86% ± 0.00 |
| Look up 1-5 pgs | 66.45% (2.3) | 86.83% ± 0.10 |
| Look up 1-6 pgs | 64.75% (2.5) | 86.91% ± 0.08 |
| ReadAgent-S 1-6 pgs | 58.53% (3.2) | 87.17% ± 0.18 |
4.3.2 NarrativeQA
NarrativeQA has the longest context length on average among the three reading comprehension datasets we choose. The dataset is divided into books (Gutenberg) and move scripts. The Gutenberg test set have 70,619 words on average, and the maximum is 343,910 words; the movie scripts test set have 29,963 on average, and the maximum is 63,957 words. As the reference answers are free-form, we evaluate based on ROUGE and the LLM Ratings (Section 4.1). The original main texts are replaced with the HTML-stripped version from SCROLLS.
在我们选择的三个阅读理解数据集中，NarrativeQA 的平均上下文长度最长。 该数据集分为书籍（Gutenberg）和电影剧本。 Gutenberg 测试集平均包含 70,619 个词，最大为 343,910 个词；电影剧本测试集平均包含 29,963 个词，最大为 63,957 个词。 由于参考答案采用自由形式，我们使用 ROUGE 和 LLM 评分（第 4.1 节）进行评估。 原始正文被替换为 SCROLLS 提供的去除 HTML 标签版本。
Because of the length of NarrativeQA articles, in order to fit the gists into the context window, we significantly expand the page size, resulting in stronger compression (Section 3.1). For example, the Gutenburg gists from the test set have 2,217 words on average and the maximum is 6,471 words, whereas the movie script gists have 2,155 words on average and the maximum is 4,511 words. Figure 3 and Figure 4 show the word statistics for the original text and the gists in Gutenberg and movie scripts respectively. The compression rate of the gists is 96.80% for Gutenberg texts and 91.98% for movie scripts. See the appendix for NarrativeQA pagination hyperparameters and more details.
由于 NarrativeQA 文章很长，为了让要点能够放入上下文窗口，我们显著增大了页面尺寸，从而产生更强的压缩（第 3.1 节）。 例如，测试集中的 Gutenberg 要点平均包含 2,217 个词，最大为 6,471 个词；电影剧本要点平均包含 2,155 个词，最大为 4,511 个词。 图 3 和 图 4 分别展示了 Gutenberg 与电影剧本中原始文本和要点的词数统计。 Gutenberg 文本要点的压缩率为 96.80%，电影剧本要点为 91.98%。 NarrativeQA 的分页超参数和更多细节见附录。
For the neural retrieval models, we use the gist memory embedding vectors rather than the page embedding vectors because the Gemini API embedding model is limited to 10,000 characters (or less than 2,000 tokens, in expectation), which is too short for embedding full pages in our NarrativeQA experiments. However, using those embedding vectors, we then return the original pages to the LLM context as normal, and use those pages as described in Section 4.2.
对于神经检索模型，我们使用要点记忆嵌入向量而不是页面嵌入向量，因为 Gemini API 嵌入模型限制为 10,000 个字符，预期不足 2,000 个 token，无法在 NarrativeQA 实验中嵌入完整页面。 不过，在使用这些嵌入向量检索后，我们仍像通常那样把原始页面返回到 LLM 上下文中，并按照第 4.2 节所述使用这些页面。
Because the Gutenberg texts and the movie scripts have significantly different distributions, we present the results separately in Table 2. ReadAgent again outperforms all the baselines across all subsets of NarrativeQA.
由于 Gutenberg 文本和电影剧本的分布显著不同，我们在 表 2 中分别展示结果。 ReadAgent 在 NarrativeQA 的所有子集上再次优于全部基线。
| Gutenberg Validation 58 docs / 1,743 questions | Gutenberg Test 177 docs / 5,207 questions | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | CR (# LU) | LR-1 | LR-2 | R-1 | R-2 | R-L | CR (# LU) | LR-1 | LR-2 | R-1 | R-2 | R-L |
| BM25 Retrieval | ||||||||||||
| Top-1 | 97.63% (1) | 39.01% | 50.14% | 0.166 | 0.061 | 0.156 | 97.42% (1) | 43.5% | 55.33% | 0.176 | 0.065 | 0.165 |
| Top-2 | 95.24% (2) | 49.34% | 60.76% | 0.203 | 0.079 | 0.191 | 94.80% (2) | 51.70% | 64.53% | 0.206 | 0.082 | 0.194 |
| Top-3 | 93.34% (3) | 52.73% | 63.68% | 0.208 | 0.080 | 0.195 | 93.02% (3) | 52.97% | 66.03% | 0.210 | 0.083 | 0.197 |
| Top-4 | 92.47% (4) | 53.59% | 64.26% | 0.211 | 0.082 | 0.197 | 92.27% (4) | 53.60% | 66.16% | 0.210 | 0.084 | 0.197 |
| Neural Retrieval with Gemini API | ||||||||||||
| Top-1 | 98.19% (1) | 34.25% | 46.53% | 0.146 | 0.051 | 0.134 | 98.14% (1) | 36.47% | 47.8% | 0.150 | 0.054 | 0.140 |
| Top-2 | 96.30% (2) | 44.69% | 54.96% | 0.180 | 0.069 | 0.167 | 96.15% (2) | 44.48% | 56.17% | 0.182 | 0.070 | 0.170 |
| Top-3 | 94.62% (3) | 46.24% | 57.31% | 0.191 | 0.077 | 0.178 | 94.42% (3) | 48.97% | 60.73% | 0.195 | 0.076 | 0.183 |
| Top-4 | 93.45% (4) | 48.59% | 59.21% | 0.196 | 0.079 | 0.184 | 93.25% (4) | 50.62% | 62.05% | 0.203 | 0.080 | 0.191 |
| GistMem | 96.89% | 55.31% | 68.22% | 0.233 | 0.091 | 0.218 | 96.80% | 55.79% | 71.19% | 0.231 | 0.092 | 0.217 |
| ReadAgent-P | ||||||||||||
| Look up 1 pg | 95.15% (0.94) | 58.92% | 71.89% | 0.244 | 0.101 | 0.230 | 94.84% (0.93) | 59.98% | 73.23% | 0.240 | 0.098 | 0.226 |
| Look up 1-2 pgs | 94.79% (1.23) | 59.84% | 72.29% | 0.239 | 0.098 | 0.224 | 94.36% (1.34) | 59.19% | 72.65% | 0.231 | 0.091 | 0.218 |
| Look up 1-3 pgs | 94.39% (1.50) | 59.84% | 71.89% | 0.240 | 0.098 | 0.226 | 94.03% (1.61) | 59.63% | 72.84% | 0.230 | 0.093 | 0.217 |
| ReadAgent-S 1-2 pgs | 94.35% (1.38) | 57.89% | 71.14% | 0.239 | 0.097 | 0.225 | 93.86% (1.46) | 60.48% | 72.48% | 0.232 | 0.095 | 0.219 |
| ReadAgent-S 1-3 pgs | 94.08% (1.57) | 58.52% | 71.49% | 0.242 | 0.098 | 0.229 | 93.67% (1.57) | 60.55% | 72.79% | 0.231 | 0.095 | 0.219 |
| Movie Validation 57 docs / 1,699 questions | Movie Test 172 docs / 5,139 questions | |||||||||||
| BM25 Retrieval | ||||||||||||
| Top-1 | 97.07% (1) | 32.67% | 42.61% | 0.156 | 0.058 | 0.144 | 96.61% (1) | 33.64% | 43.34% | 0.154 | 0.054 | 0.143 |
| Top-2 | 94.12% (2) | 39.97% | 50.21% | 0.187 | 0.070 | 0.174 | 93.81% (2) | 42.50% | 53.05% | 0.191 | 0.072 | 0.178 |
| Top-3 | 91.18% (3) | 43.61% | 53.91% | 0.198 | 0.077 | 0.185 | 91.00% (3) | 46.97% | 57.52% | 0.207 | 0.080 | 0.193 |
| Top-4 | 88.24% (4) | 46.85% | 57.62% | 0.210 | 0.084 | 0.198 | 88.19% (4) | 50.18% | 60.13% | 0.217 | 0.085 | 0.202 |
| Neural Retrieval with Gemini API | ||||||||||||
| Top-1 | 97.07% (1) | 32.02% | 41.44% | 0.153 | 0.053 | 0.142 | 96.67% (1) | 37.24% | 46.22% | 0.130 | 0.043 | 0.118 |
| Top-2 | 94.19% (2) | 43.20% | 51.38% | 0.160 | 0.057 | 0.148 | 93.90% (2) | 46.49% | 54.60% | 0.164 | 0.061 | 0.151 |
| Top-3 | 91.29% (3) | 47.56% | 56.21% | 0.176 | 0.064 | 0.163 | 91.14% (3) | 50.69% | 58.92% | 0.186 | 0.071 | 0.172 |
| Top-4 | 88.38% (4) | 49.09% | 59.33% | 0.193 | 0.075 | 0.180 | 88.36% (4) | 52.13% | 59.41% | 0.184 | 0.072 | 0.171 |
| GistMem | 92.09% | 52.56% | 64.39% | 0.242 | 0.103 | 0.227 | 91.98% | 54.68% | 64.00% | 0.248 | 0.105 | 0.234 |
| ReadAgent-P | ||||||||||||
| Look up 1 pg | 89.20% (0.99) | 53.38% | 65.57% | 0.247 | 0.106 | 0.233 | 89.22% (0.98) | 57.68% | 68.01% | 0.274 | 0.116 | 0.260 |
| Look up 1-2 pgs | 87.68% (1.52) | 54.62% | 65.63% | 0.238 | 0.098 | 0.223 | 88.10% (1.39) | 58.24% | 68.81% | 0.270 | 0.115 | 0.255 |
| Look up 1-3 pgs | 86.57% (1.91) | 54.91% | 65.86% | 0.241 | 0.099 | 0.225 | 86.73% (1.89) | 58.82% | 69.12% | 0.272 | 0.116 | 0.257 |
| ReadAgent-S 1-2 pgs | 86.36% (1.98) | 59.33% | 68.28% | 0.203 | 0.082 | 0.188 | 85.92% (1.98) | 63.33% | 72.06% | 0.214 | 0.086 | 0.199 |
| ReadAgent-S 1-3 pgs | 83.56% (2.95) | 59.45% | 68.81% | 0.210 | 0.087 | 0.195 | 83.18% (2.95) | 64.53% | 73.06% | 0.217 | 0.090 | 0.202 |
4.3.3 QMSum
QMSum consists of meeting transcripts on various topics and associated questions or instructions. We use the concatenated version of QMSum provided by SCROLLS. The transcripts tend to be quite long, ranging in length from 1,000 to 26,300 words, with an average length of about 10,000 words. Figure 5 shows the histograms of word counts for the QMSum training set. The answers are free form text, so the standard evaluation metric is ROUGE F-Measure. We additionally evaluate using our LLM Ratings (Section 4.1). See the appendix for hyperparameters and additional results.
QMSum 由不同主题的会议记录及相关问题或指令组成。 我们使用 SCROLLS 提供的 QMSum 拼接版本。 这些会议记录通常很长，长度从 1,000 到 26,300 个词不等，平均约为 10,000 个词。 图 5 展示了 QMSum 训练集词数的直方图。 答案采用自由形式文本，因此标准评估指标是 ROUGE F 值。 我们还使用第 4.1 节定义的 LLM 评分进行评估。 超参数和更多结果见附录。
| Method | CR (# LU) | LLM Rating-1 | LLM Rating-2 | ROUGE-1 | ROUGE-2 | ROUGE-L | Resp. Length |
|---|---|---|---|---|---|---|---|
| BM25 Retrieval | |||||||
| Top-1 | 95.69% (1.00) | 32.48% ± 1.65 | 63.85% ± 1.51 | 27.53 ± 0.23 | 7.00 ± 0.14 | 18.45 ± 0.16 | 48.62 ± 0.28 |
| Top-2 | 91.48% (2.00) | 29.41% ± 0.60 | 71.57% ± 1.48 | 28.85 ± 0.17 | 7.59 ± 0.08 | 19.34 ± 0.14 | 52.39 ± 0.49 |
| Top-3 | 86.93% (3.00) | 34.80% ± 1.14 | 79.53% ± 0.35 | 30.69 ± 0.17 | 8.40 ± 0.11 | 20.64 ± 0.13 | 53.59 ± 0.35 |
| Top-4 | 82.55% (4.00) | 35.66% ± 0.30 | 81.13% ± 0.35 | 31.10 ± 0.10 | 8.53 ± 0.06 | 20.36 ± 0.11 | 54.96 ± 0.42 |
| Top-5 | 78.13% (5.00) | 39.09% ± 0.92 | 84.44% ± 0.46 | 31.16 ± 0.14 | 8.52 ± 0.08 | 20.69 ± 0.03 | 54.52 ± 0.13 |
| Top-6 | 73.97% (6.00) | 37.87% ± 0.90 | 83.70% ± 0.87 | 31.06 ± 0.04 | 8.38 ± 0.06 | 20.43 ± 0.08 | 56.18 ± 0.44 |
| Neural Retrieval with Gemini API | |||||||
| Top-1 | 95.99% (1.00) | 34.80% ± 1.39 | 68.87% ± 0.62 | 27.86 ± 0.12 | 7.12 ± 0.04 | 18.76 ± 0.09 | 49.46 ± 0.23 |
| Top-2 | 92.02% (2.00) | 40.32% ± 0.92 | 81.50% ± 0.46 | 30.17 ± 0.08 | 8.03 ± 0.03 | 19.80 ± 0.08 | 55.48 ± 0.27 |
| Top-3 | 87.93% (3.00) | 40.93% ± 1.35 | 85.17% ± 1.25 | 31.36 ± 0.12 | 8.67 ± 0.10 | 20.68 ± 0.10 | 56.71 ± 0.27 |
| Top-4 | 83.71% (4.00) | 40.56% ± 0.62 | 84.31% ± 0.87 | 31.52 ± 0.11 | 8.59 ± 0.10 | 20.40 ± 0.10 | 56.47 ± 0.71 |
| Top-5 | 79.47% (5.00) | 40.20% ± 0.76 | 86.76% ± 0.60 | 31.32 ± 0.11 | 8.49 ± 0.11 | 20.49 ± 0.07 | 56.73 ± 0.91 |
| Top-6 | 75.44% (6.00) | 40.81% ± 0.52 | 87.01% ± 0.35 | 31.92 ± 0.02 | 8.73 ± 0.09 | 20.82 ± 0.05 | 58.39 ± 0.31 |
| Truncated Raw Content | |||||||
| First 6k words | 32.59% (0.00) | 14.71% ± 0.79 | 52.45% ± 0.69 | 25.42 ± 0.05 | 4.98 ± 0.09 | 16.58 ± 0.10 | 58.42 ± 0.11 |
| Last 6k words | 32.38% (0.00) | 10.42% ± 0.62 | 35.66% ± 2.46 | 20.69 ± 0.19 | 3.44 ± 0.10 | 14.13 ± 0.08 | 44.23 ± 0.11 |
| GistMem | 83.13% (0.00) | 40.20% ± 0.96 | 89.83% ± 0.76 | 31.00 ± 0.09 | 7.99 ± 0.04 | 20.15 ± 0.08 | 65.75 ± 0.20 |
| ReadAgent-P | |||||||
| Look up 1 pg | 80.00% (0.98) | 40.56% ± 0.46 | 89.46% ± 1.48 | 31.26 ± 0.09 | 8.22 ± 0.15 | 20.29 ± 0.07 | 63.78 ± 1.13 |
| Look up 1-2 pgs | 77.38% (1.71) | 39.71% ± 1.87 | 89.71% ± 0.60 | 31.11 ± 0.04 | 8.01 ± 0.15 | 20.21 ± 0.04 | 64.73 ± 1.02 |
| Look up 1-3 pgs | 75.07% (2.53) | 38.36% ± 1.21 | 89.71% ± 0.60 | 31.50 ± 0.29 | 8.15 ± 0.15 | 20.45 ± 0.24 | 63.91 ± 1.58 |
| Look up 1-4 pgs | 73.48% (3.08) | 39.95% ± 1.51 | 90.56% ± 0.35 | 31.34 ± 0.05 | 8.08 ± 0.18 | 20.26 ± 0.07 | 63.40 ± 0.79 |
| Look up 1-5 pgs | 72.29% (3.50) | 37.99% ± 0.96 | 87.75% ± 0.46 | 31.16 ± 0.10 | 8.06 ± 0.05 | 20.35 ± 0.12 | 65.22 ± 1.40 |
| Look up 1-6 pgs | 70.90% (3.97) | 39.09% ± 2.04 | 88.24% ± 0.60 | 31.50 ± 0.30 | 8.05 ± 0.13 | 20.26 ± 0.13 | 66.70 ± 0.62 |
| ReadAgent-S 1-6 pgs | 70.34% (3.55) | 46.57% ± 0.87 | 91.54% ± 0.30 | 32.90 ± 0.17 | 8.87 ± 0.23 | 21.15 ± 0.14 | 68.87 ± 0.60 |
In Table 3 and the appendix test results, we see that performance improves as the compression rate decreases, so techniques that look up more pages tend to do better than techniques that look up fewer pages. We also see that ReadAgent-S substantially outperforms ReadAgent-P (and all baselines). This performance improvement comes at a cost of up to six times as many requests in the retrieval phase. Since other datasets don't have such a strong performance improvement, we suspect that QMSum is in some sense a more challenging dataset, requiring the model to actively search through the gisted transcript to locate relevant information. This hypothesis seems reasonable, as meeting transcripts are much less structured than the documents, books, and movies found in QuALITY and NarrativeQA.
从 表 3 和附录中的测试结果可以看到，随着压缩率降低，性能会提高，因此查阅更多页面的技术往往优于查阅较少页面的技术。 我们还看到，ReadAgent-S 显著优于 ReadAgent-P 以及全部基线。 这种性能提升的代价是检索阶段的请求次数最多增加到六倍。 由于其他数据集没有如此显著的性能提升，我们推测 QMSum 在某种意义上更具挑战性，要求模型主动搜索经过要点化的会议记录来定位相关信息。 这个假设是合理的，因为会议记录的结构远不如 QuALITY 和 NarrativeQA 中的文档、书籍与电影清晰。
A large fraction of the tasks in QMSum are a request to provide a summary, rather than a concrete question about some content in the meeting. For many of these, the LLM refuses to look up any pages, instead responding with “I don't need to look up any pages. I can summarize the whole meeting based on what I already remember.”, for example. Consequently, the average number of pages looked up for ReadAgent is much lower than the maximum allowed. However, on the tasks that actually involve a question, ReadAgent tends to use most or all of the available lookup pages.
QMSum 中很大一部分任务要求提供摘要，而不是针对会议中某项内容提出具体问题。 对于其中许多任务，LLM 会拒绝查阅任何页面，例如回答：“我不需要查阅任何页面。我可以根据已经记住的内容总结整场会议。” 因此，ReadAgent 平均查阅的页数远低于允许的最大值。 不过，对于真正包含问题的任务，ReadAgent 往往会使用大部分或全部可用查阅页面。
In Table 3 and the appendix test results, the ROUGE scores by themselves don't always show a clear trend. This is because as the length of the texts increase (corresponding to the compression rates decreasing), the response lengths increase as well. Longer response lengths result in lower ROUGE precision values, which pushes down the F-Measures. Consequently, for the ROUGE scores to increase as text length increases, the improvement to recall must be more substantial than the reduction to precision. This happens to some extent, but the effect size is small.
在 表 3 和附录测试结果中，单看 ROUGE 分数并不总能发现清晰趋势。 这是因为随着文本变长，也就是压缩率降低，回答长度也会增加。 更长的回答会导致 ROUGE 精确率下降，进而拉低 F 值。 因此，要让 ROUGE 分数随文本长度增加而提高，召回率的改善必须显著大于精确率的下降。 这种情况在一定程度上确实会发生，但效应较小。
Furthermore, including gists in the text substantially increases the response length, as is the case for GistMem and all the ReadAgent approaches. This increase is in spite of the fact that all models use the same question-answering prompt, so there is no prompt difference to cause the increased response lengths. This makes it much more challenging for GistMem and ReadAgent to outperform the retrieval methods in ROUGE score. Nevertheless, ReadAgent-S manages to have the highest ROUGE scores as well as the highest LLM ratings. Because of these issues with ROUGE, we consider the LLM ratings to be more informative for comparisons between these runs. However, the LLM ratings do not make it easy to compare with results using a different LLM to rate, such as GPT, and they also do not allow for easy comparisons with other works. The same observation applies to the NarrativeQA results above.
此外，在文本中加入要点会显著增加回答长度，GistMem 和所有 ReadAgent 方法都是如此。 尽管所有模型使用相同的问答提示，不存在会导致回答变长的提示差异，这种增长仍然出现了。 这使 GistMem 和 ReadAgent 更难在 ROUGE 分数上优于检索方法。 尽管如此，ReadAgent-S 仍同时取得了最高的 ROUGE 分数和最高的 LLM 评分。 鉴于 ROUGE 存在这些问题，我们认为 LLM 评分更适合比较这些运行结果。 不过，LLM 评分不便与使用不同 LLM 评审器（例如 GPT）的结果比较，也不便与其他工作直接比较。 同样的观察也适用于上面的 NarrativeQA 结果。
4.4 Ablation Study and Analysis
Retrieval Quality. In Table 4, we compare using GistMem with neural retrieval to look up one page with using ReadAgent to look up one page. This is equivalent to replacing ReadAgent's prompt-based retrieval with neural retrieval. ReadAgent's retrieval performs better here.
检索质量。 在 表 4 中，我们比较了使用 GistMem 加神经检索查阅一页与使用 ReadAgent 查阅一页的效果。 这相当于用神经检索替换 ReadAgent 基于提示的检索。 此处 ReadAgent 的检索表现更好。
| Method | Accuracy |
|---|---|
| GistMem + Neural Retrieval Top-1 | 82.65% |
| ReadAgent-P (Look up 1 pg) | 84.13% |
Episode pagination. In this work we ask ReadAgent to decide where to pause reading and what information to store together in memory (Section 3.1), whereas in prior art, rule-based segmentation of text is typically used. We compare the two approaches with similar page length on average in Table 5 to demonstrate that it is indeed beneficial to break at pause points that LLMs consider natural (e.g. scene transitions, ends of dialogue, narrative transitions, etc).
情节分页。 在本文中，我们让 ReadAgent 决定在哪里暂停阅读以及哪些信息应共同存入记忆（第 3.1 节），而先前工作通常使用基于规则的文本分段。 我们在 表 5 中比较平均页面长度相近的两种方法，证明在 LLM 认为自然的暂停点进行切分确实有益，例如场景转换、对话结束或叙事转换。
| LLM | Uniform Length | |
|---|---|---|
| ReadAgent-P (1-5 pgs) Acc. | 86.83% | 85.71% |
The compression trade-off. Table 6 presents the empirical results of compression rate increasing as page size increases. As the compression rate decreases, the gists are more useful for answering questions directly. However, for ReadAgent with look-ups, when the initial gist compression rate gets too high, accuracy suffers.
压缩权衡。 表 6 给出了页面尺寸增大时压缩率提高的实证结果。 随着压缩率降低，要点对直接回答问题更有用。 不过，对于带查阅功能的 ReadAgent，当初始要点压缩率过高时，准确率会受损。
| GistMem | ReadAgent-P (1-5 pgs) | |||
|---|---|---|---|---|
max_words | CR | Acc | CR | Acc |
| 400 | 81.81% | 78.91% | 66.71% | 86.82% |
| 600 | 85.53% | 77.52% | 66.45% | 86.83% |
| 800 | 88.12% | 76.22% | 65.06% | 86.34% |
| 1200 | 91.38% | 73.97% | 61.77% | 85.67% |
5. Conclusion
We have presented ReadAgent, a simple interactive prompting system to mitigate the context length and context use limitations of current LLMs. ReadAgent outperforms other strong zero-shot (i.e., not trained or finetuned on the training set) baselines across standard performance metrics. These results demonstrate that LLMs are capable of generating compressed textual representations of long contexts that are useful for tasks that humans think are important, even without knowing those tasks ahead of time. They also demonstrate that LLMs are capable of reasoning interactively over such compressed representations, using them to decide what information needs to be retrieved to effectively perform a known task.
我们提出了 ReadAgent，这是一种简单的交互式提示系统，用于缓解当前 LLM 的上下文长度与上下文利用限制。 在标准性能指标上，ReadAgent 优于其他强零样本基线，也就是未在训练集上训练或微调的基线。 这些结果表明，即使事先不知道具体任务，LLM 也能够生成长上下文的压缩文本表征，而这些表征对人类认为重要的任务仍然有用。 结果还表明，LLM 能够对这类压缩表征进行交互式推理，并据此决定需要检索哪些信息，进而有效执行已知任务。
ReadAgent increases the effective context length by up to
ReadAgent 在优于传统检索技术的同时，将有效上下文长度提高了至多
Impact Statement
As ReadAgent is built atop LLMs, it naturally inherits their impacts and risks. It also makes it possible to attempt to solve new problems that current LLMs cannot tackle, due to context length limitations. It is possible that ReadAgent could cause greater harms as a consequence, just as it could improve things, depending on how it is used. One risk that we were not able to study, but that seems particularly plausible, is of an increased tendency of the LLM to hallucinate when working with gist memories rather than full text. Since many details are elided in the gist memories, if the model is called upon to perform some task that requires those details, it may generate them itself without giving any indication that is the case.
由于 ReadAgent 构建在 LLM 之上，它自然会继承 LLM 的影响与风险。 它也使我们能够尝试解决当前 LLM 因上下文长度限制而无法处理的新问题。 ReadAgent 既可能改善现状，也可能造成更大危害，具体取决于使用方式。 有一种风险我们尚未研究，但看起来尤其可能发生：与处理全文相比，LLM 在使用要点记忆时可能更容易产生幻觉。 由于要点记忆省略了许多细节，如果要求模型执行需要这些细节的任务，它可能会自行生成这些细节，却不作任何说明。