---

World Models

World ModelRLNeurIPS 2018CCF-AGoogle BrainNNAISENSEIDSIA

Ha D, Schmidhuber J. Recurrent World Models Facilitate Policy Evolution. Advances in Neural Information Processing Systems 31, 2018: 2451-2463.

https://worldmodels.github.io/

https://papers.nips.cc/paper/7512-recurrent-world-models-facilitate-policy-evolution

---

世界模型

Abstract

We explore building generative neural network models of popular reinforcement learning environments. Our world model can be trained quickly in an unsupervised manner to learn a compressed spatial and temporal representation of the environment. By using features extracted from the world model as inputs to an agent, we can train a very compact and simple policy that can solve the required task. We can even train our agent entirely inside of its own hallucinated dream generated by its world model, and transfer this policy back into the actual environment.

我们探索为常见强化学习环境构建生成式神经网络模型。 我们的世界模型可以通过无监督方式快速训练，以学习环境的压缩时空表示。 通过把从世界模型中提取的特征用作智能体输入，我们可以训练一个非常紧凑且简单的策略来解决所需任务。 我们甚至可以让智能体完全在世界模型生成的幻觉梦境中训练，再把该策略迁移回真实环境。

1. Introduction

Humans develop a mental model of the world based on what they are able to perceive with their limited senses. The decisions and actions we make are based on this internal model. Jay Wright Forrester, the father of system dynamics, described a mental model as: The image of the world around us, which we carry in our head, is just a model. Nobody in his head imagines all the world, government or country. He has only selected concepts, and relationships between them, and uses those to represent the real system.

人类根据有限感官所能感知的内容，在头脑中建立关于世界的模型。 我们作出的决定和采取的行动都基于这一内部模型。 系统动力学之父 Jay Wright Forrester 如此描述心智模型： 我们头脑中携带的周围世界图景，只是一个模型。 没有人会在头脑中想象整个世界、政府或国家。 他只选择了一些概念及其相互关系，并用它们来表示真实系统。

图1：一个世界模型，出自 Scott McCloud 的《理解漫画》。

To handle the vast amount of information that flows through our daily lives, our brain learns an abstract representation of both spatial and temporal aspects of this information. We are able to observe a scene and remember an abstract description thereof. Evidence also suggests that what we perceive at any given moment is governed by our brain's prediction of the future based on our internal model.

为了处理日常生活中涌入的海量信息，我们的大脑会学习这些信息在空间和时间两个方面的抽象表示。 我们能够观察一个场景，并记住关于它的抽象描述。 证据还表明，我们在任一时刻的感知都受大脑基于内部模型对未来所作预测的支配。

One way of understanding the predictive model inside of our brains is that it might not be about just predicting the future in general, but predicting future sensory data given our current motor actions. We are able to instinctively act on this predictive model and perform fast reflexive behaviours when we face danger, without the need to consciously plan out a course of action.

理解大脑内部预测模型的一种方式是：它可能并非只在一般意义上预测未来，而是在给定当前运动动作的情况下预测未来的感觉数据。 面对危险时，我们能够本能地依据这一预测模型行动，并迅速作出反射行为，而不必有意识地规划行动过程。

Take baseball for example. A batter has milliseconds to decide how they should swing the bat -- shorter than the time it takes for visual signals to reach our brain. The reason we are able to hit a 100 mph fastball is due to our ability to instinctively predict when and where the ball will go. For professional players, this all happens subconsciously. Their muscles reflexively swing the bat at the right time and location in line with their internal models' predictions. They can quickly act on their predictions of the future without the need to consciously roll out possible future scenarios to form a plan.

以棒球为例。 击球手只有几毫秒决定如何挥棒，这比视觉信号到达大脑所需的时间还短。 我们之所以能够击中时速 100 英里的快球，是因为我们可以本能地预测球将在何时到达何处。 对于职业球员，这一切都在潜意识中发生。 他们的肌肉会按照内部模型的预测，在正确的时间和位置反射性地挥棒。 他们可以迅速依据对未来的预测行动，而无需有意识地展开各种可能的未来情景来制定计划。

图2：我们所看到的内容基于大脑对未来的预测。

In many reinforcement learning (RL) problems, an artificial agent also benefits from having a good representation of past and present states, and a good predictive model of the future, preferably a powerful predictive model implemented on a general purpose computer such as a recurrent neural network (RNN).

在许多强化学习（RL）问题中，人工智能体同样受益于对过去和当前状态的良好表示，以及对未来的良好预测模型；理想情况下，这应当是一个在通用计算机上实现的强大预测模型，例如循环神经网络（RNN）。

图3：我们为 OpenAI Gym 环境构建概率生成模型。基于 RNN 的世界模型使用真实游戏环境中记录的观测进行训练；训练完成后，它可以模拟完整环境，并供智能体在其中训练。

Large RNNs are highly expressive models that can learn rich spatial and temporal representations of data. However, many model-free RL methods in the literature often only use small neural networks with few parameters. The RL algorithm is often bottlenecked by the credit assignment problem, which makes it hard for traditional RL algorithms to learn millions of weights of a large model, hence in practice, smaller networks are used as they iterate faster to a good policy during training.

大型 RNN 是表达能力很强的模型，能够学习丰富的数据时空表示。 然而，文献中的许多无模型强化学习方法通常只使用参数很少的小型神经网络。 强化学习算法往往受信用分配问题限制，传统强化学习算法因而难以学习大型模型中的数百万个权重；实践中通常采用较小的网络，因为它们能在训练期间更快迭代至良好策略。

Ideally, we would like to be able to efficiently train large RNN-based agents. The backpropagation algorithm can be used to train large neural networks efficiently. In this work we look at training a large neural network to tackle RL tasks, by dividing the agent into a large world model and a small controller model. We first train a large neural network to learn a model of the agent's world in an unsupervised manner, and then train the smaller controller model to learn to perform a task using this world model. A small controller lets the training algorithm focus on the credit assignment problem on a small search space, while not sacrificing capacity and expressiveness via the larger world model. By training the agent through the lens of its world model, we show that it can learn a highly compact policy to perform its task. Typical model-free RL models have in the order of ${10}^3$ to ${10}^6$ model parameters. We look at training models in the order of ${10}^7$ parameters, which is still rather small compared to state-of-the-art deep learning models with ${10}^8$ to even ${10}^9$ parameters. In principle, the procedure described in this article can take advantage of these larger networks if we wanted to use them.

理想情况下，我们希望能够高效训练基于大型 RNN 的智能体。 反向传播算法可以高效训练大型神经网络。 在本文中，我们把智能体划分为大型世界模型和小型控制器模型，以此研究如何训练大型神经网络解决强化学习任务。 我们首先以无监督方式训练大型神经网络来学习智能体所处世界的模型，然后训练较小的控制器模型，使其学会利用该世界模型执行任务。 小型控制器让训练算法可以在较小的搜索空间中专注处理信用分配问题，同时又通过更大的世界模型保留容量和表达能力。 我们表明，透过世界模型训练智能体，可以使其学会执行任务的高度紧凑策略。 典型无模型强化学习模型的参数量约为 ${10}^3$ 至 ${10}^6$。 我们研究训练约含 ${10}^7$ 个参数的模型；与拥有 ${10}^8$ 乃至 ${10}^9$ 个参数的先进深度学习模型相比，这仍然相当小。 原则上，如有需要，本文所述流程可以利用这些更大的网络。

Although there is a large body of research relating to model-based reinforcement learning, this article is not meant to be a review of the current state of the field. Instead, the goal of this article is to distill several key concepts from a series of papers 1990--2015 on combinations of RNN-based world models and controllers. We will also discuss other related works in the literature that share similar ideas of learning a world model and training an agent using this model.

尽管已有大量研究涉及基于模型的强化学习，但本文无意综述该领域的当前状态。 相反，本文旨在提炼 1990 至 2015 年间一系列有关基于 RNN 的世界模型与控制器组合研究中的若干关键概念。 我们还将讨论文献中其他具有相似思想的相关工作，即学习世界模型，并使用该模型训练智能体。

In this article, we present a simplified framework that we can use to experimentally demonstrate some of the key concepts from these papers, and also suggest further insights to effectively apply these ideas to various RL environments. We use similar terminology and notation as On Learning to Think: Algorithmic Information Theory for Novel Combinations of RL Controllers and RNN World Models when describing our methodology and experiments.

本文提出一个简化框架，用于通过实验展示这些论文中的若干关键概念，并进一步说明如何把这些思想有效应用于各种强化学习环境。 在描述方法和实验时，我们采用与《On Learning to Think: Algorithmic Information Theory for Novel Combinations of RL Controllers and RNN World Models》相近的术语与记号。

2. Agent Model

We present a simple model inspired by our own cognitive system. In this model, our agent has a visual sensory component that compresses what it sees into a small representative code. It also has a memory component that makes predictions about future codes based on historical information. Finally, our agent has a decision-making component that decides what actions to take based only on the representations created by its vision and memory components.

我们提出一个受人类认知系统启发的简单模型。 在该模型中，智能体拥有视觉感知组件，把它看到的内容压缩为小型表示编码。 它还拥有记忆组件，根据历史信息预测未来编码。 最后，智能体包含决策组件，仅依据视觉与记忆组件产生的表示决定要采取的动作。

图4：智能体由三个紧密协作的组件构成：视觉（V）、记忆（M）和控制器（C）。

2.1 VAE (V) Model

The environment provides our agent with a high dimensional input observation at each time step. This input is usually a 2D image frame that is part of a video sequence. The role of the V model is to learn an abstract, compressed representation of each observed input frame.

环境在每个时间步为智能体提供高维输入观测。 该输入通常是视频序列中的一帧二维图像。 V 模型的作用是学习每个观测输入帧的抽象压缩表示。

图5：变分自编码器（VAE）流程图。

Here, we use a simple Variational Autoencoder as our V model to compress each image frame into a small latent vector $z$.

这里，我们使用简单的变分自编码器作为 V 模型，把每个图像帧压缩为小型潜向量 $z$。

2.2 MDN-RNN (M) Model

While it is the role of the V model to compress what the agent sees at each time frame, we also want to compress what happens over time. For this purpose, the role of the M model is to predict the future. The M model serves as a predictive model of the future $z$ vectors that V is expected to produce. Since many complex environments are stochastic in nature, we train our RNN to output a probability density function $p(z)$ instead of a deterministic prediction of $z$.

V 模型负责压缩智能体在每一时刻看到的内容，而我们还希望压缩随时间发生的事情。 为此，M 模型的作用是预测未来。 M 模型充当未来 $z$ 向量的预测模型，这些向量是 V 预计会产生的表示。 由于许多复杂环境本质上具有随机性，我们训练 RNN 输出概率密度函数 $p(z)$，而非对 $z$ 作确定性预测。

图6：带有混合密度网络输出层的 RNN。MDN 输出高斯混合分布的参数，用于采样下一潜向量 z 的预测。

In our approach, we approximate $p(z)$ as a mixture of Gaussian distribution, and train the RNN to output the probability distribution of the next latent vector $z_{t+1}$ given the current and past information made available to it.

在我们的方法中，$p(z)$ 被近似为高斯混合分布，并训练 RNN 在给定当前与历史信息的条件下，输出下一潜向量 $z_{t+1}$ 的概率分布。

More specifically, the RNN will model $P(z_{t+1}\mid a_t,z_t,h_t)$, where $a_t$ is the action taken at time $t$ and $h_t$ is the hidden state of the RNN at time $t$. During sampling, we can adjust a temperature parameter $\tau$ to control model uncertainty, as done in SketchRNN -- we will find adjusting $\tau$ to be useful for training our controller later on.

更具体地说，RNN 建模 $P(z_{t+1}\mid a_t,z_t,h_t)$，其中 $a_t$ 是时间 $t$ 采取的动作，$h_t$ 是 RNN 在时间 $t$ 的隐藏状态。 采样时，我们可以像 SketchRNN 那样调节温度参数 $\tau$ 来控制模型不确定性；后文将看到，调节 $\tau$ 对训练控制器很有用。

图7：SketchRNN 是使用 MDN-RNN 预测草图下一笔画的示例。我们使用类似模型预测下一潜向量 z_t。

This approach is known as a Mixture Density Network combined with a RNN (MDN-RNN), and has been applied in the past for sequence generation problems such as generating handwriting and sketches.

这种方法称为混合密度网络与 RNN 的组合（MDN-RNN），过去已用于手写体和草图生成等序列生成问题。

2.3 Controller (C) Model

The Controller (C) model is responsible for determining the course of actions to take in order to maximize the expected cumulative reward of the agent during a rollout of the environment. In our experiments, we deliberately make C as simple and small as possible, and trained separately from V and M, so that most of our agent's complexity resides in the world model (V and M).

控制器（C）模型负责确定应采取的动作过程，以最大化智能体在一次环境轨迹中的期望累积奖励。 在实验中，我们有意让 C 尽可能简单且小型，并将它与 V、M 分开训练，使智能体的大部分复杂性都位于世界模型（V 和 M）中。

C is a simple single layer linear model that maps $z_t$ and $h_t$ directly to action $a_t$ at each time step:

C 是一个简单的单层线性模型，在每个时间步把 $z_t$ 和 $h_t$ 直接映射为动作 $a_t$：

$$a_t=W_c[z_t{h}_t]+b_c$$

In this linear model, $W_c$ and $b_c$ are the weight matrix and bias vector that maps the concatenated input vector $[z_t{h}_t]$ to the output action vector $a_t$.

在这个线性模型中，$W_c$ 和 $b_c$ 分别是权重矩阵与偏置向量，它们把拼接输入向量 $[z_t{h}_t]$ 映射为输出动作向量 $a_t$。

2.4 Putting V, M, and C Together

The following flow diagram illustrates how V, M, and C interacts with the environment:

下列流程图展示 V、M 与 C 如何同环境交互：

图8：智能体模型流程图。每个时间步 t 的原始观测先由 V 处理得到 z_t；C 接收 z_t 与 M 的隐藏状态 h_t，输出运动控制动作 a_t；M 再使用当前 z_t 和 a_t 更新隐藏状态，产生供下一时间步使用的 h_{t+1}。

Below is the pseudocode for how our agent model is used in the OpenAI Gym environment:

下面给出智能体模型在 OpenAI Gym 环境中的使用伪代码：

def rollout(controller):
  ''' env, rnn, vae are '''
  ''' global variables  '''
  obs = env.reset()
  h = rnn.initial_state()
  done = False
  cumulative_reward = 0
  while not done:
    z = vae.encode(obs)
    a = controller.action([z, h])
    obs, reward, done = env.step(a)
    cumulative_reward += reward
    h = rnn.forward([a, z, h])
  return cumulative_reward

Running this function on a given controller C will return the cumulative reward during a rollout.

对给定的 controller C 运行该函数，将返回一次轨迹中的累积奖励。

This minimal design for C also offers important practical benefits. Advances in deep learning provided us with the tools to train large, sophisticated models efficiently, provided we can define a well-behaved, differentiable loss function. Our V and M models are designed to be trained efficiently with the backpropagation algorithm using modern GPU accelerators, so we would like most of the model's complexity, and model parameters to reside in V and M. The number of parameters of C, a linear model, is minimal in comparison. This choice allows us to explore more unconventional ways to train C -- for example, even using evolution strategies (ES) to tackle more challenging RL tasks where the credit assignment problem is difficult.

这种极简的 C 设计还带来了重要的实践优势。 只要能够定义性质良好的可微损失函数，深度学习的发展便为我们提供了高效训练大型复杂模型的工具。 V 和 M 模型被设计为可利用现代 GPU 加速器与反向传播算法进行高效训练，因此我们希望模型的大部分复杂性和参数都位于 V 与 M 中。 相比之下，线性模型 C 的参数量极少。 这一选择使我们可以探索更非常规的 C 训练方法，例如使用进化策略（ES）处理信用分配困难、更加具有挑战性的强化学习任务。

To optimize the parameters of C, we chose the Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) as our optimization algorithm since it is known to work well for solution spaces of up to a few thousand parameters. We evolve parameters of C on a single machine with multiple CPU cores running multiple rollouts of the environment in parallel.

为优化 C 的参数，我们选择协方差矩阵自适应进化策略（CMA-ES）作为优化算法，因为已知它能很好处理最多数千个参数的解空间。 我们在一台多核 CPU 机器上演化 C 的参数，并行运行多次环境轨迹。

For more specific information about the models, training procedures, and environments used in our experiments, please refer to the Appendix section.

有关实验所用模型、训练流程和环境的更多具体信息，请参阅附录。

3. Car Racing Experiment

In this section, we describe how we can train the Agent model described earlier to solve a car racing task. To our knowledge, our agent is the first known solution to achieve the score required to solve this task. We find this task interesting because although it is not difficult to train an agent to wobble around randomly generated tracks and obtain a mediocre score, CarRacing-v0 defines solving as getting average reward of 900 over 100 consecutive trials, which means the agent can only afford very few driving mistakes.

本节介绍如何训练前述智能体模型来解决赛车任务。 据我们所知，该智能体是首个达到任务解决分数要求的已知方案。 这一任务很有意思：虽然训练智能体在随机生成的赛道上摇摆前进并取得一般分数并不困难，但 CarRacing-v0 把连续 100 次试验的平均奖励达到 900 定义为解决任务，这意味着智能体只能出现极少的驾驶失误。

3.1 World Model for Feature Extraction

A predictive world model can help us extract useful representations of space and time. By using these features as inputs of a controller, we can train a compact and minimal controller to perform a continuous control task, such as learning to drive from pixel inputs for a top-down car racing environment called CarRacing-v0.

预测式世界模型可以帮助我们提取有用的时空表示。 通过把这些特征作为控制器输入，我们可以训练紧凑的极简控制器来执行连续控制任务，例如从像素输入中学习在名为 CarRacing-v0 的俯视赛车环境中驾驶。

图9：智能体学习在 CarRacing-v0 中导航。

In this environment, the tracks are randomly generated for each trial, and our agent is rewarded for visiting as many tiles as possible in the least amount of time. The agent controls three continuous actions: steering left/right, acceleration, and brake.

在该环境中，每次试验的赛道都会随机生成；智能体以尽可能短的时间访问尽可能多的赛道块即可获得奖励。 智能体控制三个连续动作：左右转向、加速和制动。

To train our V model, we first collect a dataset of 10,000 random rollouts of the environment. We have first an agent acting randomly to explore the environment multiple times, and record the random actions $a_t$ taken and the resulting observations from the environment. We use this dataset to train V to learn a latent space of each frame observed. We train our VAE to encode each frame into low dimensional latent vector $z$ by minimizing the difference between a given frame and the reconstructed version of the frame produced by the decoder from $z$.

为训练 V 模型，我们首先收集包含 10,000 次随机环境轨迹的数据集。 我们先让智能体以随机动作多次探索环境，并记录所采取的随机动作 $a_t$ 及由此得到的环境观测。 我们使用该数据集训练 V，使其学习每个观测帧的潜空间。 通过最小化给定帧与解码器从 $z$ 生成的重建帧之间的差异，我们训练 VAE 把每一帧编码为低维潜向量 $z$。

We can now use our trained V model to pre-process each frame at time $t$ into $z_t$ to train our M model. Using this pre-processed data, along with the recorded random actions $a_t$ taken, our MDN-RNN can now be trained to model $P(z_{t+1}\mid a_t,z_t,h_t)$ as a mixture of Gaussians. In principle, we can train both models together in an end-to-end manner, although we found that training each separately is more practical, and also achieves satisfactory results. Training each model only required less than an hour of computation time on a single GPU. We can also train individual VAE and MDN-RNN models without having to exhaustively tune hyperparameters.

现在，我们可以使用训练好的 V 模型把时间 $t$ 的每一帧预处理为 $z_t$，用于训练 M 模型。 利用这些预处理数据与记录的随机动作 $a_t$，即可训练 MDN-RNN，把 $P(z_{t+1}\mid a_t,z_t,h_t)$ 建模为高斯混合分布。 原则上可以端到端联合训练两个模型，但我们发现分别训练更加实用，也能取得令人满意的结果。 在单个 GPU 上训练每个模型所需的计算时间都不到一小时。 我们还可以单独训练 VAE 和 MDN-RNN，而无需穷尽式调节超参数。

In this experiment, the world model (V and M) has no knowledge about the actual reward signals from the environment. Its task is simply to compress and predict the sequence of image frames observed. Only the Controller (C) Model has access to the reward information from the environment. Since there are a mere 867 parameters inside the linear controller model, evolutionary algorithms such as CMA-ES are well suited for this optimization task.

在该实验中，世界模型（V 和 M）并不知道环境中的真实奖励信号。 它的任务仅仅是压缩并预测观测到的图像帧序列。 只有控制器（C）模型能够访问环境奖励信息。 由于线性控制器模型仅含 867 个参数，CMA-ES 等进化算法非常适合这一优化任务。

We can use the VAE to reconstruct each frame using $z_t$ at each time step to visualize the quality of the information the agent actually sees during a rollout. The figure below is a VAE model trained on screenshots from CarRacing-v0.

我们可以让 VAE 在每个时间步使用 $z_t$ 重建当前帧，以可视化智能体在一次轨迹中实际看到的信息质量。 下图展示了使用 CarRacing-v0 截图训练的 VAE 模型。

图10：尽管有损压缩过程会丢失细节，潜向量 z 仍能捕获每个图像帧的核心内容。

In the online version of this article, one can load randomly chosen screenshots to be encoded into a small latent vector $z$, which is used to reconstruct the original screenshot. One can also experiment with adjusting the values of the $z$ vector using the slider bars to see how it affects the reconstruction, or randomize $z$ to observe the space of possible screenshots.

在本文在线版本中，读者可以载入随机选择的截图，将其编码为小型潜向量 $z$，再用该向量重建原始截图。 读者还可以使用滑块调节 $z$ 向量的数值，观察其对重建的影响，或者随机化 $z$ 来观察可能截图的空间。

3.2 Procedure

To summarize the Car Racing experiment, below are the steps taken:

1. Collect 10,000 rollouts from a random policy.

2. Train VAE (V) to encode frames into $z\in {\mathcal{R}}^{32}$.

3. Train MDN-RNN (M) to model $P(z_{t+1}\mid a_t,z_t,h_t)$.

4. Define Controller (C) as $a_t=W_c[z_t{h}_t]+b_c$.

5. Use CMA-ES to solve for a $W_c$ and $b_c$ that maximizes the expected cumulative reward.

概括而言，赛车实验采用以下步骤：

1. 从随机策略收集 10,000 次轨迹。

2. 训练 VAE（V），把图像帧编码为 $z\in {\mathcal{R}}^{32}$。

3. 训练 MDN-RNN（M）建模 $P(z_{t+1}\mid a_t,z_t,h_t)$。

4. 把控制器（C）定义为 $a_t=W_c[z_t{h}_t]+b_c$。

5. 使用 CMA-ES 求解使期望累积奖励最大化的 $W_c$ 和 $b_c$。

Model	Parameter Count
VAE	4,348,547
MDN-RNN	422,368
Controller	867

3.3 Experiment Results

V Model Only

Training an agent to drive is not a difficult task if we have a good representation of the observation. Previous works have shown that with a good set of hand-engineered information about the observation, such as LIDAR information, angles, positions and velocities, one can easily train a small feed-forward network to take this hand-engineered input and output a satisfactory navigation policy. For this reason, we first want to test our agent by handicapping C to only have access to V but not M, so we define our controller as $a_t=W_c{z}_t+b_c$.

如果拥有良好的观测表示，训练智能体驾驶并非难事。 既有工作表明，只要具备一组良好的人工设计观测信息，例如激光雷达信息、角度、位置和速度，就可以轻松训练小型前馈网络接收这些人工输入，并输出令人满意的导航策略。 因此，我们首先通过限制 C 只能访问 V 而不能访问 M 来测试智能体，并把控制器定义为 $a_t=W_c{z}_t+b_c$。

图11：限制控制器只能看到 z_t 而不能看到 h_t，会导致摇摆且不稳定的驾驶行为。

Although the agent is still able to navigate the race track in this setting, we notice it wobbles around and misses the tracks on sharper corners. This handicapped agent achieved an average score of $632\pm 251$ over 100 random trials, in line with the performance of other agents on OpenAI Gym's leaderboard and traditional Deep RL methods such as A3C. Adding a hidden layer to C's policy network helps to improve the results to $788\pm 141$, but not quite enough to solve this environment.

尽管在这种设置下智能体仍能沿赛道行驶，但我们注意到它会左右摇摆，并在较急的弯道上偏离赛道。 这个受限智能体在 100 次随机试验中的平均分为 $632\pm 251$，与 OpenAI Gym 排行榜上的其他智能体以及 A3C 等传统深度强化学习方法表现相当。 在 C 的策略网络中增加隐藏层可以把结果提高到 $788\pm 141$，但仍不足以解决该环境。

Full World Model (V and M)

The representation $z_t$ provided by our V model only captures a representation at a moment in time and does not have much predictive power. In contrast, M is trained to do one thing, and to do it really well, which is to predict $z_{t+1}$. Since M's prediction of $z_{t+1}$ is produced from the RNN's hidden state $h_t$ at time $t$, this vector is a good candidate for the set of learned features we can give to our agent. Combining $z_t$ with $h_t$ gives our controller C a good representation of both the current observation, and what to expect in the future.

V 模型提供的表示 $z_t$ 只捕获某一时刻的表示，并不具备很强的预测能力。 相比之下，M 只接受一项任务的训练，并把它做得很好，即预测 $z_{t+1}$。 由于 M 对 $z_{t+1}$ 的预测来自 RNN 在时间 $t$ 的隐藏状态 $h_t$，该向量很适合作为提供给智能体的一组学习特征。 把 $z_t$ 与 $h_t$ 结合起来，可为控制器 C 同时提供当前观测以及未来预期的良好表示。

图12：让控制器同时访问 z_t 和 h_t 后，驾驶会更加稳定。

We see that allowing the agent to access the both $z_t$ and $h_t$ greatly improves its driving capability. The driving is more stable, and the agent is able to seemingly attack the sharp corners effectively. Furthermore, we see that in making these fast reflexive driving decisions during a car race, the agent does not need to plan ahead and roll out hypothetical scenarios of the future. Since $h_t$ contain information about the probability distribution of the future, the agent can just query the RNN instinctively to guide its action decisions. Like a seasoned Formula One driver or the baseball player discussed earlier, the agent can instinctively predict when and where to navigate in the heat of the moment.

可以看到，让智能体同时访问 $z_t$ 和 $h_t$ 会显著提升其驾驶能力。 驾驶更加稳定，智能体似乎也能有效应对急弯。 此外，在赛车过程中作出这些快速反射式驾驶决策时，智能体无需提前规划并展开假想的未来情景。 由于 $h_t$ 包含未来概率分布的信息，智能体只需本能地查询 RNN，便可引导其动作决策。 就像经验丰富的一级方程式赛车手或前文讨论的棒球运动员一样，智能体可以在紧要关头本能地预测应在何时向何处行驶。

表1：CarRacing-v0 中各种方法取得的分数。

Method	Avg. Score
DQN	343 ± 18
A3C (continuous)	591 ± 45
A3C (discrete)	652 ± 10
ceobillionaire (Gym Leaderboard)	838 ± 11
V model	632 ± 251
V model with hidden layer	788 ± 141
Full World Model	906 ± 21

Our agent is able to achieve a score of $906\pm 21$ over 100 random trials, effectively solving the task and obtaining new state of the art results. Previous attempts using Deep RL methods obtained average scores of 591--652 range, and the best reported solution on the leaderboard obtained an average score of $838\pm 11$ over 100 random trials. Traditional Deep RL methods often require pre-processing of each frame, such as employing edge-detection, in addition to stacking a few recent frames into the input. In contrast, our world model takes in a stream of raw RGB pixel images and directly learns a spatial-temporal representation. To our knowledge, our method is the first reported solution to solve this task.

我们的智能体在 100 次随机试验中取得 $906\pm 21$ 分，有效解决了该任务并刷新了先进结果。 此前使用深度强化学习方法的尝试取得 591 至 652 的平均分区间，排行榜上报告的最佳方案在 100 次随机试验中平均得分为 $838\pm 11$。 传统深度强化学习方法通常需要预处理每一帧，例如采用边缘检测，还要把最近若干帧堆叠为输入。 相比之下，我们的世界模型接收原始 RGB 像素图像流，并直接学习时空表示。 据我们所知，我们的方法是首个被报告能够解决该任务的方案。

3.4 Car Racing Dreams

Since our world model is able to model the future, we are also able to have it come up with hypothetical car racing scenarios on its own. We can ask it to produce the probability distribution of $z_{t+1}$ given the current states, sample a $z_{t+1}$ and use this sample as the real observation. We can put our trained C back into this hallucinated environment generated by M. The following image from an interactive demo in the online version of this article shows how our world model can be used to hallucinate the car racing environment:

由于世界模型能够建模未来，我们还可以让它自行构想假想的赛车情景。 我们可以要求它在给定当前状态时生成 $z_{t+1}$ 的概率分布，采样一个 $z_{t+1}$，并把该样本用作真实观测。 我们可以把训练好的 C 放回 M 生成的这一幻觉环境中。 下面来自本文在线版交互演示的图像展示了如何使用世界模型生成赛车环境幻觉：

图13：智能体在自己的梦境世界中驾驶。训练好的策略被部署到 MDN-RNN 生成、VAE 解码器渲染的虚假环境中；交互演示允许覆盖智能体动作并调节温度 τ，以控制 M 生成环境的不确定性。

4. VizDoom Experiment

4.1 Learning Inside of a Dream

We have just seen that a policy learned inside of the real environment appears to somewhat function inside of the dream environment. This begs the question -- can we train our agent to learn inside of its own dream, and transfer this policy back to the actual environment?

我们刚刚看到，在真实环境中学到的策略似乎也能在梦境环境中发挥一定作用。 这引出了一个问题：我们能否让智能体在自己的梦境中学习，再把该策略迁移回真实环境？

If our world model is sufficiently accurate for its purpose, and complete enough for the problem at hand, we should be able to substitute the actual environment with this world model. After all, our agent does not directly observe the reality, but only sees what the world model lets it see. In this experiment, we train an agent inside the hallucination generated by its world model trained to mimic a VizDoom environment.

如果世界模型对其用途而言足够准确，并且对当前问题足够完整，我们就应当能够用该世界模型替代真实环境。 毕竟，智能体并不直接观察现实，只能看到世界模型允许它看到的内容。 在该实验中，我们训练世界模型模拟 VizDoom 环境，再让智能体在该模型生成的幻觉中训练。

图14：最终智能体解决 VizDoom: Take Cover。

The agent must learn to avoid fireballs shot by monsters from the other side of the room with the sole intent of killing the agent. There are no explicit rewards in this environment, so to mimic natural selection, the cumulative reward can be defined to be the number of time steps the agent manages to stay alive during a rollout. Each rollout of the environment runs for a maximum of 2100 time steps ($\sim$ 60 seconds), and the task is considered solved if the average survival time over 100 consecutive rollouts is greater than 750 time steps ($\sim$ 20 seconds).

智能体必须学会躲避房间另一端怪物发射的火球，而这些火球的唯一目的就是杀死智能体。 该环境没有显式奖励，因此为了模拟自然选择，可以把累积奖励定义为智能体在一次轨迹中成功存活的时间步数。 每次环境轨迹最多运行 2100 个时间步（$\sim$ 60 秒）；若连续 100 次轨迹的平均存活时间超过 750 个时间步（$\sim$ 20 秒），则认为任务已被解决。

4.2 Procedure

The setup of our VizDoom experiment is largely the same as the Car Racing task, except for a few key differences. In the Car Racing task, M is only trained to model the next $z_t$. Since we want to build a world model we can train our agent in, our M model here will also predict whether the agent dies in the next frame (as a binary event $don{e}_t$, or $d_t$ for short), in addition to the next frame $z_t$.

VizDoom 实验的设置与赛车任务大体相同，只存在少数关键差异。 在赛车任务中，M 只接受预测下一 $z_t$ 的训练。 由于我们希望构建一个可供智能体训练的世界模型，这里的 M 模型除了预测下一帧 $z_t$，还会预测智能体是否会在下一帧死亡，并把它表示为二元事件 $don{e}_t$，简称 $d_t$。

Since the M model can predict the $done$ state in addition to the next observation, we now have all of the ingredients needed to make a full RL environment. We first build an OpenAI Gym environment interface by wrapping a gym.Env interface over our M if it were a real Gym environment, and then train our agent inside of this virtual environment instead of using the actual environment.

由于 M 模型除了下一观测之外还能预测 $done$ 状态，我们现在已经具备构建完整强化学习环境所需的全部要素。 我们首先在 M 外封装一个 gym.Env 接口，把它当作真实 Gym 环境来构建 OpenAI Gym 环境接口，然后在这个虚拟环境中而非真实环境中训练智能体。

In this simulation, we do not need the V model to encode any real pixel frames during the hallucination process, so our agent will therefore only train entirely in a latent space environment. This has many advantages as we will see.

在该模拟中，幻觉过程不需要 V 模型编码任何真实像素帧，因此智能体完全在潜空间环境中训练。 正如后文将看到的，这具有许多优势。

This virtual environment has an identical interface to the real environment, so after the agent learns a satisfactory policy in the virtual environment, we can easily deploy this policy back into the actual environment to see how well the policy transfers over.

虚拟环境与真实环境具有相同接口，因此智能体在虚拟环境中学到令人满意的策略后，我们可以轻松把该策略部署回真实环境，观察其迁移效果。

To summarize the Take Cover experiment, below are the steps taken:

1. Collect 10,000 rollouts from a random policy.

2. Train VAE (V) to encode each frame into a latent vector $z\in {\mathcal{R}}^{64}$, and use V to convert the images collected from (1) into the latent space representation.

3. Train MDN-RNN (M) to model $P(z_{t+1},d_{t+1}\mid a_t,z_t,h_t)$.

4. Define Controller (C) as $a_t=W_c[z_t{h}_t]$.

5. Use CMA-ES to solve for a $W_c$ that maximizes the expected survival time inside the virtual environment.

6. Use learned policy from (5) on actual environment.

概括而言，Take Cover 实验采用以下步骤：

1. 从随机策略收集 10,000 次轨迹。

2. 训练 VAE（V），把每一帧编码为潜向量 $z\in {\mathcal{R}}^{64}$，并使用 V 把步骤（1）收集的图像转换为潜空间表示。

3. 训练 MDN-RNN（M）建模 $P(z_{t+1},d_{t+1}\mid a_t,z_t,h_t)$。

4. 把控制器（C）定义为 $a_t=W_c[z_t{h}_t]$。

5. 使用 CMA-ES 求解使虚拟环境中期望存活时间最大化的 $W_c$。

6. 把步骤（5）学到的策略用于真实环境。

Model	Parameter Count
VAE	4,446,915
MDN-RNN	1,678,785
Controller	1,088

4.3 Training Inside of the Dream

After some training, our controller learns to navigate around the dream environment and escape from deadly fireballs launched by monsters generated by M. Our agent achieved a score of $\sim$ 900 time steps in the virtual environment.

经过一段时间的训练，控制器学会了在梦境环境中移动，并躲避 M 生成的怪物所发射的致命火球。 智能体在虚拟环境中取得约 $\sim$ 900 个时间步的分数。

图15：智能体发现一种躲避幻觉火球的策略。在本文在线版中，读者可以在演示中与该环境交互。

Here, our RNN-based world model is trained to mimic a complete game environment designed by human programmers. By learning only from raw image data collected from random episodes, it learns how to simulate the essential aspects of the game -- such as the game logic, enemy behaviour, physics, and also the 3D graphics rendering.

这里，基于 RNN 的世界模型接受训练，以模拟由人类程序员设计的完整游戏环境。 它仅从随机回合中收集的原始图像数据学习，便学会了模拟游戏的关键方面，例如游戏逻辑、敌人行为、物理过程以及三维图形渲染。

For instance, if the agent selects the left action, the M model learns to move the agent to the left and adjust its internal representation of the game states accordingly. It also learns to block the agent from moving beyond the walls on both sides of the level if the agent attempts to move too far in either direction. Occasionally, the M model needs to keep track of multiple fireballs being shot from several different monsters and coherently move them along in their intended directions. It must also detect whether the agent has been killed by one of these fireballs.

例如，当智能体选择向左动作时，M 模型会学习把智能体向左移动，并相应调整其对游戏状态的内部表示。 它还会学习在智能体试图向任一方向移动过远时，阻止其越过关卡两侧的墙壁。 有时，M 模型需要同时追踪多个不同怪物发射的多个火球，并让它们沿预定方向连贯移动。 它还必须检测智能体是否被其中某个火球杀死。

Unlike the actual game environment, however, we note that it is possible to add extra uncertainty into the virtual environment, thus making the game more challenging in the dream environment. We can do this by increasing the temperature $\tau$ parameter during the sampling process of $z_{t+1}$. By increasing the uncertainty, our dream environment becomes more difficult compared to the actual environment. The fireballs may move more randomly in a less predictable path compared to the actual game. Sometimes the agent may even die due to sheer misfortune, without explanation.

然而，与真实游戏环境不同，我们可以向虚拟环境中加入额外不确定性，从而提高梦境环境中的游戏难度。 这可以通过在 $z_{t+1}$ 的采样过程中增大温度参数 $\tau$ 来实现。 提高不确定性后，梦境环境会比真实环境更加困难。 与真实游戏相比，火球可能以更随机、更难预测的路径移动。 有时，智能体甚至可能纯粹因为运气不佳而莫名死亡。

We find agents that perform well in higher temperature settings generally perform better in the normal setting. In fact, increasing $\tau$ helps prevent our controller from taking advantage of the imperfections of our world model -- we will discuss this in more depth later on.

我们发现，在较高温度设置下表现良好的智能体，通常在正常设置下也表现更好。 事实上，增大 $\tau$ 有助于防止控制器利用世界模型中的缺陷；后文将对此作更深入讨论。

4.4 Transfer Policy to Actual Environment

图16：把在梦境 RNN 环境中学到的策略部署回真实 VizDoom 环境。

We took the agent trained inside of the virtual environment and tested its performance on the original VizDoom scenario. The score over 100 random consecutive trials is $\sim$ 1100 time steps, far beyond the required score of 750 time steps, and also much higher than the score obtained inside the more difficult virtual environment.

我们把在虚拟环境中训练的智能体放到原始 VizDoom 场景中测试其性能。 连续 100 次随机试验的得分约为 $\sim$ 1100 个时间步，远超 750 个时间步的要求，也显著高于在更困难的虚拟环境中取得的分数。

图17：本文在线版中的交互式 Doom VAE。

We see that even though the V model is not able to capture all of the details of each frame correctly, for instance, getting the number of monsters correct, the agent is still able to use the learned policy to navigate in the real environment. As the virtual environment cannot even keep track of the exact number of monsters in the first place, an agent that is able to survive the noisier and uncertain virtual nightmare environment will thrive in the original, cleaner environment.

可以看到，尽管 V 模型无法正确捕获每一帧的全部细节，例如无法准确表示怪物数量，智能体仍能使用所学策略在真实环境中行动。 由于虚拟环境原本就无法追踪怪物的确切数量，能够在噪声更多、更加不确定的虚拟噩梦环境中存活的智能体，自然可以在原始且更干净的环境中表现出色。

4.5 Cheating the World Model

In our childhood, we may have encountered ways to exploit video games in ways that were not intended by the original game designer. Players discover ways to collect unlimited lives or health, and by taking advantage of these exploits, they can easily complete an otherwise difficult game. However, in the process of doing so, they may have forfeited the opportunity to learn the skill required to master the game as intended by the game designer.

童年时，我们可能遇到过以原游戏设计者未曾预期的方式利用电子游戏漏洞的方法。 玩家会发现获取无限生命或生命值的方法，并利用这些漏洞轻松通关原本困难的游戏。 然而，在此过程中，他们可能也放弃了学习游戏设计者原本希望玩家掌握的通关技能。

For instance, in our initial experiments, we noticed that our agent discovered an adversarial policy to move around in such a way so that the monsters in this virtual environment governed by the M model never shoots a single fireball during some rollouts. Even when there are signs of a fireball forming, the agent will move in a way to extinguish the fireballs magically as if it has superpowers in the environment.

例如，在最初的实验中，我们注意到智能体发现了一种对抗性移动策略，使 M 模型控制的虚拟环境中的怪物在某些轨迹里从不发射任何火球。 即使出现火球正在形成的迹象，智能体也会以某种方式移动，像在环境中拥有超能力一般神奇地熄灭火球。

Because our world model is only an approximate probabilistic model of the environment, it will occasionally generate trajectories that do not follow the laws governing the actual environment. As we saw previously, even the number of monsters on the other side of the room in the actual environment is not exactly reproduced by the world model. Like a child who learns that objects in the air usually fall to the ground, the child might also imagine unrealistic superheroes who fly across the sky. For this reason, our world model will be exploitable by the controller, even if in the actual environment such exploits do not exist.

由于世界模型只是环境的近似概率模型，它有时会生成不遵循真实环境规律的轨迹。 正如前文所见，即使是真实环境中房间另一端的怪物数量，世界模型也无法精确复现。 就像孩子学到空中的物体通常会落向地面，却也可能想象出在天空飞行的不现实超级英雄。 因此，即便真实环境中不存在这些漏洞，控制器仍可能利用世界模型。

And since we are using the M model to generate a virtual dream environment for our agent, we are also giving the controller access to all of the hidden states of M. This is essentially granting our agent access to all of the internal states and memory of the game engine, rather than only the game observations that the player gets to see. Therefore our agent can efficiently explore ways to directly manipulate the hidden states of the game engine in its quest to maximize its expected cumulative reward. The weakness of this approach of learning a policy inside a learned dynamics model is that our agent can easily find an adversarial policy that can fool our dynamics model -- it'll find a policy that looks good under our dynamics model, but will fail in the actual environment, usually because it visits states where the model is wrong because they are away from the training distribution.

由于我们使用 M 模型为智能体生成虚拟梦境环境，也就同时允许控制器访问 M 的全部隐藏状态。 这实质上让智能体能够访问游戏引擎的全部内部状态和记忆，而不再局限于玩家可以看到的游戏观测。 因此，为了最大化期望累积奖励，智能体可以高效探索直接操纵游戏引擎隐藏状态的方法。 在学习所得动力学模型中学习策略的弱点在于，智能体很容易找到欺骗动力学模型的对抗策略：该策略在动力学模型下看似很好，却会在真实环境中失败；这通常是因为它访问了偏离训练分布、模型预测错误的状态。

图18：智能体发现一种对抗策略，可在某些轨迹中让已发射的火球自动熄灭。

This weakness could be the reason that many previous works that learn dynamics models of RL environments but do not actually use those models to fully replace the actual environments. Like in the M model proposed in earlier work, the dynamics model is a deterministic model, making the model easily exploitable by the agent if it is not perfect. Using Bayesian models, as in PILCO, helps to address this issue with the uncertainty estimates to some extent, however, they do not fully solve the problem. Recent work combines the model-based approach with traditional model-free RL training by first initializing the policy network with the learned policy, but must subsequently rely on model-free methods to fine-tune this policy in the actual environment.

这一弱点或许可以解释，为何许多既有工作会学习强化学习环境的动力学模型，却并未真正使用这些模型完全替代真实环境。 与早期工作提出的 M 模型类似，动力学模型是确定性模型；只要它不完美，智能体就很容易利用其中的缺陷。 像 PILCO 那样使用贝叶斯模型，可以借助不确定性估计在一定程度上缓解这一问题，但仍无法彻底解决。 近期工作把基于模型的方法与传统无模型强化学习训练结合起来，先用学到的策略初始化策略网络，但随后仍必须依赖无模型方法在真实环境中微调该策略。

In Learning to Think, it is acceptable that the RNN M is not always a reliable predictor. A (potentially evolution-based) RNN C can in principle learn to ignore a flawed M, or exploit certain useful parts of M for arbitrary computational purposes including hierarchical planning etc. This is not what we do here though -- our present approach is still closer to some of the older systems, where a RNN M is used to predict and plan ahead step by step. Unlike this early work, however, we use evolution for C (like in Learning to Think) rather than traditional RL combined with RNNs, which has the advantage of both simplicity and generality.

在 Learning to Think 中，RNN M 不必始终是可靠的预测器。 原则上，基于 RNN 的 C，可能还采用进化方法，可以学会忽略有缺陷的 M，或者利用 M 的某些有用部分实现包括层次规划在内的任意计算目的。 不过，这并不是本文采用的方法；我们当前的方法仍更接近一些较早的系统，其中 RNN M 被用于逐步预测和提前规划。 然而，与这些早期工作不同，我们像 Learning to Think 那样使用进化方法训练 C，而非把传统强化学习与 RNN 结合起来，因此同时具备简单性和通用性的优势。

To make it more difficult for our C model to exploit deficiencies of the M model, we chose to use the MDN-RNN as the dynamics model, which models the distribution of possible outcomes in the actual environment, rather than merely predicting a deterministic future. Even if the actual environment is deterministic, the MDN-RNN would in effect approximate it as a stochastic environment. This has the advantage of allowing us to train our C model inside a more stochastic version of any environment -- we can simply adjust the temperature parameter $\tau$ to control the amount of randomness in the M model, hence controlling the tradeoff between realism and exploitability.

为提高 C 模型利用 M 模型缺陷的难度，我们选择 MDN-RNN 作为动力学模型；它建模真实环境中各种可能结果的分布，而不仅仅预测确定性未来。 即使真实环境是确定性的，MDN-RNN 实际上也会把它近似为随机环境。 这样做的优点是，我们可以在任何环境的更随机版本中训练 C 模型；只需调节温度参数 $\tau$，即可控制 M 模型中的随机程度，从而控制真实性与可利用性之间的权衡。

Using a mixture of Gaussian model may seem like overkill given that the latent space encoded with the VAE model is just a single diagonal Gaussian distribution. However, the discrete modes in a mixture density model is useful for environments with random discrete events, such as whether a monster decides to shoot a fireball or stay put. While a single diagonal Gaussian might be sufficient to encode individual frames, a RNN with a mixture density output layer makes it easier to model the logic behind a more complicated environment with discrete random states.

考虑到 VAE 模型编码的潜空间只是单个对角高斯分布，使用高斯混合模型似乎有些大材小用。 然而，混合密度模型中的离散模态对包含随机离散事件的环境很有用，例如怪物是决定发射火球还是留在原地。 单个对角高斯分布或许足以编码单帧，但带混合密度输出层的 RNN 更容易建模具有离散随机状态的复杂环境背后的逻辑。

For instance, if we set the temperature parameter to a very low value of $\tau =0.1$, effectively training our C model with a M model that is almost identical to a deterministic LSTM, the monsters inside this dream environment fail to shoot fireballs, no matter what the agent does, due to mode collapse. The M model is not able to jump to another mode in the mixture of Gaussian model where fireballs are formed and shot. Whatever policy learned inside of this dream will achieve a perfect score of 2100 most of the time, but will obviously fail when unleashed into the harsh reality of the actual world, underperforming even a random policy.

例如，如果把温度参数设为很低的 $\tau =0.1$，实际上相当于使用几乎等同于确定性 LSTM 的 M 模型训练 C；由于模态坍缩，无论智能体做什么，梦境环境中的怪物都无法发射火球。 M 模型无法跳转到高斯混合模型中形成并发射火球的另一个模态。 在该梦境中学到的任何策略，大多数时候都会取得 2100 的满分；但一旦被释放到严酷的真实世界中，显然会失败，甚至不如随机策略。

Note again, however, that the simpler and more robust approach in Learning to Think does not insist on using M for step by step planning. Instead, C can learn to use M's subroutines (parts of M's weight matrix) for arbitrary computational purposes but can also learn to ignore M when M is useless and when ignoring M yields better performance. Nevertheless, at least in our present C--M variant, M's predictions are essential for teaching C, more like in some of the early C--M systems, but combined with evolution or black box optimization.

不过需要再次指出，Learning to Think 中更简单、更稳健的方法并不坚持使用 M 进行逐步规划。 相反，C 可以学习把 M 的子程序，即 M 权重矩阵的一部分，用于任意计算目的；当 M 无用且忽略 M 能取得更好性能时，C 也可以学会忽略 M。 尽管如此，至少在当前 C--M 变体中，M 的预测对于教会 C 仍至关重要；它更接近某些早期 C--M 系统，只是结合了进化或黑盒优化。

By making the temperature $\tau$ an adjustable parameter of the M model, we can see the effect of training the C model on hallucinated virtual environments with different levels of uncertainty, and see how well they transfer over to the actual environment. We experimented with varying the temperature of the virtual environment and observing the resulting average score over 100 random rollouts of the actual environment after training the agent inside of the virtual environment with a given temperature:

把温度 $\tau$ 设为 M 模型的可调参数后，我们可以观察在不同不确定性水平的幻觉虚拟环境中训练 C 模型所产生的影响，以及所得策略迁移到真实环境的效果。 我们改变虚拟环境的温度，在给定温度下于虚拟环境中训练智能体后，观察其在真实环境的 100 次随机轨迹中的平均分数：

表2：不同温度设置下的 Take Cover 分数。

Temperature τ	Virtual Score	Actual Score
0.10	2086 ± 140	193 ± 58
0.50	2060 ± 277	196 ± 50
1.00	1145 ± 690	868 ± 511
1.15	918 ± 546	1092 ± 556
1.30	732 ± 269	753 ± 139
Random Policy	N/A	210 ± 108
Gym Leader	N/A	820 ± 58

We see that while increasing the temperature of the M model makes it more difficult for the C model to find adversarial policies, increasing it too much will make the virtual environment too difficult for the agent to learn anything, hence in practice it is a hyperparameter we can tune. The temperature also affects the types of strategies the agent discovers. For example, although the best score obtained is $1092\pm 556$ with $\tau =1.15$, increasing $\tau$ a notch to 1.30 results in a lower score but at the same time a less risky strategy with a lower variance of returns. For comparison, the best score on the OpenAI Gym leaderboard is $820\pm 58$.

可以看到，提高 M 模型的温度虽然会增加 C 模型寻找对抗策略的难度，但温度过高也会让虚拟环境困难到智能体无法学到任何东西，因此实践中它是一个可调超参数。 温度还会影响智能体发现的策略类型。 例如，尽管 $\tau =1.15$ 时取得了 $1092\pm 556$ 的最佳分数，但把 $\tau$ 略微提高至 1.30 后，分数有所下降，却同时得到风险更低、回报方差更小的策略。 作为对比，OpenAI Gym 排行榜上的最佳分数为 $820\pm 58$。

5. Iterative Training Procedure

In our experiments, the tasks are relatively simple, so a reasonable world model can be trained using a dataset collected from a random policy. But what if our environments become more sophisticated? In any difficult environment, only parts of the world are made available to the agent only after it learns how to strategically navigate through its world.

实验中的任务相对简单，因此使用随机策略收集的数据集就能训练出合理的世界模型。 但如果环境变得更加复杂呢？ 在任何困难环境中，只有当智能体学会如何有策略地探索其世界后，世界的某些部分才会向它开放。

For more complicated tasks, an iterative training procedure is required. We need our agent to be able to explore its world, and constantly collect new observations so that its world model can be improved and refined over time. An iterative training procedure is as follows:

1. Initialize M, C with random model parameters.

2. Rollout to actual environment $N$ times. Save all actions $a_t$ and observations $x_t$ during rollouts to storage.

3. Train M to model $P(x_{t+1},r_{t+1},a_{t+1},d_{t+1}\mid x_t,a_t,h_t)$ and train C to optimize expected rewards inside of M.

4. Go back to (2) if task has not been completed.

更复杂的任务需要迭代训练流程。 智能体需要能够探索其世界并持续收集新观测，使世界模型随时间不断改进与完善。 迭代训练流程如下：

1. 使用随机模型参数初始化 M 和 C。

2. 在真实环境中运行 $N$ 次轨迹，把轨迹中的所有动作 $a_t$ 和观测 $x_t$ 保存到存储中。

3. 训练 M 建模 $P(x_{t+1},r_{t+1},a_{t+1},d_{t+1}\mid x_t,a_t,h_t)$，并训练 C 在 M 内优化期望奖励。

4. 若任务尚未完成，则返回步骤（2）。

We have shown that one iteration of this training loop was enough to solve simple tasks. For more difficult tasks, we need our controller in Step 2 to actively explore parts of the environment that is beneficial to improve its world model. An exciting research direction is to look at ways to incorporate artificial curiosity and intrinsic motivation and information seeking abilities in an agent to encourage novel exploration. In particular, we can augment the reward function based on improvement in compression quality.

我们已经表明，该训练循环迭代一次便足以解决简单任务。 对于更困难的任务，步骤 2 中的控制器需要主动探索有助于改进世界模型的环境部分。 一个令人兴奋的研究方向，是探索如何把人工好奇心、内在动机和信息寻求能力融入智能体，以鼓励新颖探索。 具体而言，我们可以根据压缩质量的改善来增强奖励函数。

In the present approach, since M is a MDN-RNN that models a probability distribution for the next frame, if it does a poor job, then it means the agent has encountered parts of the world that it is not familiar with. Therefore we can adapt and reuse M's training loss function to encourage curiosity. By flipping the sign of M's loss function in the actual environment, the agent will be encouraged to explore parts of the world that it is not familiar with. The new data it collects may improve the world model.

在当前方法中，M 是对下一帧概率分布进行建模的 MDN-RNN；如果它表现不佳，就意味着智能体遇到了世界中不熟悉的部分。 因此，我们可以调整并复用 M 的训练损失函数来鼓励好奇心。 通过在真实环境中反转 M 损失函数的符号，智能体将受到鼓励去探索自己不熟悉的世界部分。 它收集的新数据可能会改进世界模型。

The iterative training procedure requires the M model to not only predict the next observation $x$ and $done$, but also predict the action and reward for the next time step. This may be required for more difficult tasks. For instance, if our agent needs to learn complex motor skills to walk around its environment, the world model will learn to imitate its own C model that has already learned to walk. After difficult motor skills, such as walking, is absorbed into a large world model with lots of capacity, the smaller C model can rely on the motor skills already absorbed by the world model and focus on learning more higher level skills to navigate itself using the motor skills it had already learned.

迭代训练流程要求 M 模型不仅预测下一观测 $x$ 和 $done$，还要预测下一时间步的动作与奖励。 更困难的任务可能需要这一能力。 例如，如果智能体需要学习复杂运动技能以在环境中行走，世界模型将学习模仿已经学会行走的自身 C 模型。 当行走等困难运动技能被吸收到容量充足的大型世界模型后，较小的 C 模型便可以依赖世界模型已吸收的运动技能，专注学习如何使用已经掌握的运动技能进行导航等更高层技能。

图19：信息如何成为记忆。

An interesting connection to the neuroscience literature is the work on hippocampal replay that examines how the brain replays recent experiences when an animal rests or sleeps. Replaying recent experiences plays an important role in memory consolidation -- where hippocampus-dependent memories become independent of the hippocampus over a period of time. As Replay Comes of Age puts it, replay is less like dreaming and more like thought. We invite readers to read Replay Comes of Age for a detailed overview of replay from a neuroscience perspective with connections to theoretical reinforcement learning.

与神经科学文献的一个有趣联系，是有关海马体回放的研究，它考察动物休息或睡眠时大脑如何回放近期经历。 回放近期经历在记忆巩固中发挥重要作用；在这一过程中，依赖海马体的记忆会逐渐变得不再依赖海马体。 正如 Replay Comes of Age 所言，回放与其说像做梦，不如说更像思考。 我们建议读者阅读 Replay Comes of Age，从神经科学视角深入了解回放及其与理论强化学习的联系。

Iterative training could allow the C--M model to develop a natural hierarchical way to learn. Recent works about self-play in RL and PowerPlay also explores methods that lead to a natural curriculum learning, and we feel this is one of the more exciting research areas of reinforcement learning.

迭代训练可以让 C--M 模型形成自然的层次化学习方式。 近期有关强化学习自博弈和 PowerPlay 的工作也在探索产生自然课程学习的方法；我们认为，这是强化学习中更令人兴奋的研究领域之一。

6. Related Work

There is extensive literature on learning a dynamics model, and using this model to train a policy. Many concepts first explored in the 1980s for feed-forward neural networks (FNNs) and in the 1990s for RNNs laid some of the groundwork for Learning to Think. The more recent PILCO is a probabilistic model-based search policy method designed to solve difficult control problems. Using data collected from the environment, PILCO uses a Gaussian process (GP) model to learn the system dynamics, and then uses this model to sample many trajectories in order to train a controller to perform a desired task, such as swinging up a pendulum, or riding a unicycle.

已有大量文献研究如何学习动力学模型并使用该模型训练策略。 许多最早在 20 世纪 80 年代针对前馈神经网络（FNN）、在 90 年代针对 RNN 探索的概念，为 Learning to Think 奠定了部分基础。 较新的 PILCO 是一种基于概率模型的策略搜索方法，旨在解决困难控制问题。 PILCO 使用从环境中收集的数据，通过高斯过程（GP）模型学习系统动力学，再用该模型采样大量轨迹，以训练控制器执行所需任务，例如把钟摆摆起或驾驶独轮车。

图20：带有内部 RNN 世界模型的控制器。

While Gaussian processes work well with a small set of low dimension data, their computational complexity makes them difficult to scale up to model a large history of high dimensional observations. Other recent works use Bayesian neural networks instead of GPs to learn a dynamics model. These methods have demonstrated promising results on challenging control tasks, where the states are known and well defined, and the observation is relatively low dimensional. Here we are interested in modelling dynamics observed from high dimensional visual data where our input is a sequence of raw pixel frames.

高斯过程很适合处理少量低维数据，但其计算复杂度使它难以扩展到对大量高维历史观测进行建模。 其他近期工作使用贝叶斯神经网络而非 GP 来学习动力学模型。 这些方法在具有挑战性的控制任务上展示了良好结果，其中状态已知且定义明确，观测维度也相对较低。 本文关注从高维视觉数据中观察到的动力学，输入是一系列原始像素帧。

In robotic control applications, the ability to learn the dynamics of a system from observing only camera-based video inputs is a challenging but important problem. Early work on RL for active vision trained an FNN to take the current image frame of a video sequence to predict the next frame, and use this predictive model to train a fovea-shifting control network trying to find targets in a visual scene. To get around the difficulty of training a dynamical model to learn directly from high-dimensional pixel images, researchers explored using neural networks to first learn a compressed representation of the video frames. Recent work along these lines was able to train controllers using the bottleneck hidden layer of an autoencoder as low-dimensional feature vectors to control a pendulum from pixel inputs. Learning a model of the dynamics from a compressed latent space enable RL algorithms to be much more data-efficient. We invite readers to watch Finn's lecture on Model-Based RL to learn more.

在机器人控制应用中，仅通过观察相机视频输入来学习系统动力学，是一个具有挑战性但很重要的问题。 早期主动视觉强化学习工作训练 FNN，使用视频序列的当前图像帧预测下一帧，并利用这一预测模型训练中央凹转移控制网络，在视觉场景中寻找目标。 为绕开直接从高维像素图像训练动力学模型的困难，研究者探索了先使用神经网络学习视频帧压缩表示的方法。 沿这一路线的近期工作能够使用自编码器瓶颈隐藏层作为低维特征向量来训练控制器，从像素输入控制钟摆。 从压缩潜空间学习动力学模型，可以显著提高强化学习算法的数据效率。 我们建议读者观看 Finn 关于基于模型强化学习的讲座，以进一步了解该主题。

Video game environments are also popular in model-based RL research as a testbed for new ideas. Earlier work used a feed-forward convolutional neural network (CNN) to learn a forward simulation model of a video game. Learning to predict how different actions affect future states in the environment is useful for game-play agents, since if our agent can predict what happens in the future given its current state and action, it can simply select the best action that suits its goal. This has been demonstrated not only in early work (when compute was a million times more expensive than today) but also in recent studies on several competitive VizDoom environments.

电子游戏环境也是基于模型强化学习研究中测试新思想的常用平台。 早期工作使用前馈卷积神经网络（CNN）学习电子游戏的前向模拟模型。 学习预测不同动作如何影响环境未来状态，对游戏智能体很有帮助；如果智能体能够在给定当前状态和动作时预测未来会发生什么，它就可以直接选择最符合目标的动作。 这一点不仅在计算成本比今天高一百万倍的早期工作中得到证明，也在多个竞技 VizDoom 环境上的近期研究中得到验证。

The works mentioned above use FNNs to predict the next video frame. We may want to use models that can capture longer term time dependencies. RNNs are powerful models suitable for sequence modelling. In a lecture called Hallucination with RNNs, Graves demonstrated the ability of RNNs to learn a probabilistic model of Atari game environments. He trained RNNs to learn the structure of such a game and then showed that they can hallucinate similar game levels on its own.

上述工作使用 FNN 预测下一视频帧。 我们可能希望使用能够捕获更长期时间依赖关系的模型。 RNN 是适合序列建模的强大模型。 在名为 Hallucination with RNNs 的讲座中，Graves 展示了 RNN 学习 Atari 游戏环境概率模型的能力。 他训练 RNN 学习这类游戏的结构，并展示这些模型可以自行生成类似游戏关卡的幻觉。

Using RNNs to develop internal models to reason about the future has been explored as early as 1990 in a paper called Making the World Differentiable, and then further explored in subsequent work. A more recent paper called Learning to Think presented a unifying framework for building a RNN-based general problem solver that can learn a world model of its environment and also learn to reason about the future using this model. Subsequent works have used RNN-based models to generate many frames into the future, and also as an internal model to reason about the future.

早在 1990 年，论文 Making the World Differentiable 就已探索使用 RNN 构建内部模型以推理未来，后续工作又作了进一步研究。 较新的论文 Learning to Think 提出一个统一框架，用于构建基于 RNN 的通用问题求解器，使其既能学习环境世界模型，又能使用该模型学习推理未来。 后续工作使用基于 RNN 的模型生成未来多个帧，也把它用作推理未来的内部模型。

In this work, we used evolution strategies to train our controller, as it offers many benefits. For instance, we only need to provide the optimizer with the final cumulative reward, rather than the entire history. ES is also easy to parallelize -- we can launch many instances of rollout with different solutions to many workers and quickly compute a set of cumulative rewards in parallel. Recent works have confirmed that ES is a viable alternative to traditional Deep RL methods on many strong baselines.

本文使用进化策略训练控制器，因为它具有许多优势。 例如，我们只需向优化器提供最终累积奖励，而不必提供完整历史。 ES 也很容易并行化：我们可以在多个工作进程上使用不同解启动多个 rollout 实例，并快速并行计算一组累积奖励。 近期工作已在许多强基线上证实，ES 是传统深度强化学习方法的可行替代方案。

Before the popularity of Deep RL methods, evolution-based algorithms have been shown to be effective at solving RL tasks. Evolution-based algorithms have even been able to solve difficult RL tasks from high dimensional pixel inputs. More recent works combine VAE and ES, which is similar to our approach.

在深度强化学习方法普及之前，基于进化的算法就已被证明能够有效解决强化学习任务。 基于进化的算法甚至能够从高维像素输入解决困难的强化学习任务。 较新的工作把 VAE 与 ES 结合起来，这与我们的方法相似。

7. Discussion

图21：1990 年绘制的基于 RNN 的控制器与环境交互示意图。

We have demonstrated the possibility of training an agent to perform tasks entirely inside of its simulated latent space dream world. This approach offers many practical benefits. For instance, running computationally intensive game engines require using heavy compute resources for rendering the game states into image frames, or calculating physics not immediately relevant to the game. We may not want to waste cycles training an agent in the actual environment, but instead train the agent as many times as we want inside its simulated environment. Training agents in the real world is even more expensive, so world models that are trained incrementally to simulate reality may prove to be useful for transferring policies back to the real world. Our approach may complement sim2real approaches.

我们已经证明，完全在模拟潜空间梦境世界中训练智能体执行任务是可行的。 这种方法具有许多实践优势。 例如，运行计算密集型游戏引擎需要大量计算资源把游戏状态渲染为图像帧，或计算与游戏并非直接相关的物理过程。 我们或许不希望浪费计算周期在真实环境中训练智能体，而是希望在模拟环境中按需进行任意多次训练。 在现实世界中训练智能体的成本更高，因此通过增量训练来模拟现实的世界模型，可能有助于把策略迁移回现实世界。 我们的方法可以与 sim2real 方法形成互补。

Furthermore, we can take advantage of deep learning frameworks to accelerate our world model simulations using GPUs in a distributed environment. The benefit of implementing the world model as a fully differentiable recurrent computation graph also means that we may be able to train our agents in the dream directly using the backpropagation algorithm to fine-tune its policy to maximize an objective function.

此外，我们可以利用深度学习框架，在分布式环境中使用 GPU 加速世界模型模拟。 把世界模型实现为完全可微的循环计算图还有另一项优势：我们或许能够直接在梦境中使用反向传播算法训练智能体，微调其策略以最大化目标函数。

The choice of using a VAE for the V model and training it as a standalone model also has its limitations, since it may encode parts of the observations that are not relevant to a task. After all, unsupervised learning cannot, by definition, know what will be useful for the task at hand. For instance, it reproduced unimportant detailed brick tile patterns on the side walls in the Doom environment, but failed to reproduce task-relevant tiles on the road in the Car Racing environment. By training together with a M model that predicts rewards, the VAE may learn to focus on task-relevant areas of the image, but the tradeoff here is that we may not be able to reuse the VAE effectively for new tasks without retraining.

选择 VAE 作为 V 模型并单独训练也有其局限，因为它可能编码与任务无关的观测部分。 毕竟，根据定义，无监督学习无法知道哪些内容对当前任务有用。 例如，它复现了 Doom 环境侧墙上并不重要的砖块细节图案，却未能复现赛车环境道路上与任务相关的赛道块。 若与预测奖励的 M 模型联合训练，VAE 或许可以学会关注图像中与任务相关的区域，但代价是若不重新训练，我们可能无法在新任务中有效复用该 VAE。

Learning task-relevant features has connections to neuroscience as well. Primary sensory neurons are released from inhibition when rewards are received, which suggests that they generally learn task-relevant features, rather than just any features, at least in adulthood.

学习与任务相关的特征也与神经科学存在联系。 获得奖励时，初级感觉神经元会解除抑制；这表明至少在成年期，它们通常学习与任务相关的特征，而非任意特征。

Another concern is the limited capacity of our world model. While modern storage devices can store large amounts of historical data generated using the iterative training procedure, our LSTM-based world model may not be able to store all of the recorded information inside its weight connections. While the human brain can hold decades and even centuries of memories to some resolution, our neural networks trained with backpropagation have more limited capacity and suffer from issues such as catastrophic forgetting. Future work may explore replacing the small MDN-RNN network with higher capacity models, or incorporating an external memory module, if we want our agent to learn to explore more complicated worlds.

另一个问题是世界模型容量有限。 现代存储设备可以保存迭代训练流程产生的大量历史数据，但基于 LSTM 的世界模型可能无法在其权重连接中存储全部记录信息。 人脑能够以一定分辨率保存数十年乃至数百年的记忆，而使用反向传播训练的神经网络容量更加有限，并且会遭受灾难性遗忘等问题。 如果希望智能体学会探索更复杂的世界，未来工作可以研究使用容量更大的模型替代小型 MDN-RNN 网络，或引入外部记忆模块。

Like early RNN-based C--M systems, ours simulates possible futures time step by time step, without profiting from human-like hierarchical planning or abstract reasoning, which often ignores irrelevant spatial-temporal details. However, the more general Learning To Think approach is not limited to this rather naive approach. Instead it allows a recurrent C to learn to address subroutines of the recurrent M, and reuse them for problem solving in arbitrary computable ways, e.g., through hierarchical planning or other kinds of exploiting parts of M's program-like weight matrix. A recent One Big Net extension of the C--M approach collapses C and M into a single network, and uses PowerPlay-like behavioural replay (where the behaviour of a teacher net is compressed into a student net) to avoid forgetting old prediction and control skills when learning new ones. Experiments with those more general approaches are left for future work.

与早期基于 RNN 的 C--M 系统一样，我们的系统逐时间步模拟可能的未来，无法利用人类式层次规划或抽象推理，而后两者通常会忽略无关的时空细节。 然而，更通用的 Learning To Think 方法并不局限于这种相当天真的做法。 它允许循环 C 学习调用循环 M 的子程序，并以任意可计算方式复用它们来解决问题，例如进行层次规划，或以其他方式利用 M 中类似程序的权重矩阵部分。 近期 C--M 方法的 One Big Net 扩展把 C 和 M 合并为单个网络，并使用类似 PowerPlay 的行为回放，其中教师网络的行为被压缩到学生网络中，以避免在学习新技能时遗忘旧的预测与控制技能。 对这些更通用方法的实验留待未来工作。