---

Mastering Diverse Domains through World Models

World ModelRLNature 2025Google DeepMindUniversity of Toronto

Hafner D, Pasukonis J, Ba J, et al. Mastering Diverse Domains through World Models. arXiv:2301.04104, 2023.

Hafner D, Pasukonis J, Ba J, et al. Mastering diverse control tasks through world models. Nature, 2025, 640: 647-653.

https://danijar.com/project/dreamerv3/

https://github.com/danijar/dreamerv3

---

通过世界模型掌握多样化领域

Abstract

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

开发一种能够学会解决广泛应用领域任务的通用算法，一直是人工智能领域的一项根本挑战。 尽管当前的强化学习算法可以直接应用于与其开发目标相似的任务，但要将其配置到新的应用领域，仍需要大量人类专业知识和实验。 我们提出 DreamerV3，这是一种只需一套配置便能在 150 多项多样化任务上超越专用方法的通用算法。 Dreamer 学习环境模型，并通过想象未来情景来改进自身行为。 基于归一化、平衡和变换的稳健性技术，使其能够跨领域稳定学习。 Dreamer 无需任何专门调整，便成为首个在没有人类数据或课程学习的情况下从零开始在 Minecraft 中收集到钻石的算法。 这一成就长期以来被视为人工智能领域的一项重大挑战，需要智能体在开放世界中根据像素输入和稀疏奖励探索具有远见的策略。 我们的工作使智能体无需大量实验即可解决具有挑战性的控制问题，从而让强化学习得到广泛应用。

图1：基准汇总。a：Dreamer 在所有领域使用固定超参数，跨多种基准和数据预算超越经过调优的专家算法，并显著优于高质量 PPO 实现。b：Dreamer 无需专门调整即可在 Minecraft 中从零开始学习获得钻石。

1. Introduction

Reinforcement learning has enabled computers to solve tasks through interaction, such as surpassing humans in the games of Go and Dota. It is also a key component for improving large language models beyond what is demonstrated in their pretraining data. While PPO has become a standard algorithm in the field of reinforcement learning, more specialized algorithms are often employed to achieve higher performance. These specialized algorithms target the unique challenges posed by different application domains, such as continuous control, discrete actions, sparse rewards, image inputs, spatial environments, and board games. However, applying reinforcement learning algorithms to sufficiently new tasks---such as moving from video games to robotics tasks---requires substantial effort, expertise, and computational resources for tweaking the hyperparameters of the algorithm. This brittleness poses a bottleneck in applying reinforcement learning to new problems and also limits the applicability of reinforcement learning to computationally expensive models or tasks where tuning is prohibitive. Creating a general algorithm that learns to master new domains without having to be reconfigured has been a central challenge in artificial intelligence and would open up reinforcement learning to a wide range of practical applications.

强化学习使计算机能够通过交互解决任务，例如在围棋和 Dota 游戏中超越人类。 它也是改进大语言模型、使其能力超越预训练数据中所展示水平的关键组成部分。 尽管 PPO 已成为强化学习领域的标准算法，但为了获得更高性能，研究者通常会采用更加专用的算法。 这些专用算法针对不同应用领域中的独特挑战，例如连续控制、离散动作、稀疏奖励、图像输入、空间环境和棋盘游戏。 然而，将强化学习算法应用于差异足够大的新任务，例如从电子游戏转向机器人任务，需要投入大量精力、专业知识和计算资源来调整算法超参数。 这种脆弱性成为强化学习应用于新问题时的瓶颈，也限制了强化学习在计算成本高昂的模型或难以进行调优的任务中的适用性。 构建一种无需重新配置便能学会掌握新领域的通用算法，一直是人工智能领域的核心挑战；它将使强化学习能够用于广泛的实际应用。

We present Dreamer, a general algorithm that outperforms specialized expert algorithms across a wide range of domains while using fixed hyperparameters, making reinforcement learning readily applicable to new problems. The algorithm is based on the idea of learning a world model that equips the agent with rich perception and the ability to imagine the future. The world model predicts the outcomes of potential actions, a critic neural network judges the value of each outcome, and an actor neural network chooses actions to reach the best outcomes. Although intuitively appealing, robustly learning and leveraging world models to achieve strong task performance has been an open problem. Dreamer overcomes this challenge through a range of robustness techniques based on normalization, balancing, and transformations. We observe robust learning not only across over 150 tasks from the domains summarized in Figure 2, but also across model sizes and training budgets, offering a predictable way to increase performance. Notably, larger model sizes not only achieve higher scores but also require less interaction to solve a task.

我们提出 Dreamer，这是一种使用固定超参数便能在广泛领域中超越专用专家算法的通用算法，使强化学习可以直接应用于新问题。 该算法建立在学习世界模型这一思想之上，使智能体具备丰富的感知能力和想象未来的能力。 世界模型预测潜在动作的结果，评论家神经网络判断每种结果的价值，而行动者神经网络选择能够达到最佳结果的动作。 尽管这一思想直观而有吸引力，但如何稳健地学习并利用世界模型来获得强大的任务性能，一直是一个开放问题。 Dreamer 通过一系列基于归一化、平衡和变换的稳健性技术克服了这一挑战。 我们不仅在 图2 所概括领域中的 150 多项任务上观察到稳健学习，而且在不同模型规模和训练预算下也观察到了这一点，从而提供了一种可预测的性能提升方式。 值得注意的是，更大的模型不仅能取得更高分数，也只需更少的交互即可解决任务。

Control Suite

Atari

ProcGen

DMLab

Minecraft

图2：实验中使用的多样化视觉领域。Dreamer 成功覆盖机器人运动与操作、Atari 游戏、程序生成的 ProcGen 关卡、需要时空推理的 DMLab 任务以及复杂且无限的 Minecraft 世界；我们还在非视觉领域中评估 Dreamer。

To push the boundaries of reinforcement learning, we consider the popular video game Minecraft that has become a focal point of research in recent years, with international competitions held for developing algorithms that autonomously learn to collect diamonds in Minecraft. The MineRL Diamond Competitions were held in 2019, 2020, and 2021 and provided a dataset of human expert trajectories. Competitions in the following years focused on a wide range of tasks. Solving this problem without human data has been widely recognized as a substantial challenge for artificial intelligence because of the sparse rewards, exploration difficulty, long time horizons, and the procedural diversity of this open world game. Due to these obstacles, previous approaches resorted to using human expert data and domain-specific curricula. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch.

为了拓展强化学习的能力边界，我们研究了近年来成为研究焦点的流行电子游戏 Minecraft；相关国际竞赛旨在开发能够自主学习在 Minecraft 中收集钻石的算法。 MineRL Diamond 竞赛于 2019、2020 和 2021 年举行，并提供了人类专家轨迹数据集。 随后几年的竞赛则聚焦于广泛的任务。 由于该开放世界游戏具有稀疏奖励、探索困难、时间跨度长和程序生成多样性，在没有人类数据的情况下解决这一问题，被广泛认为是人工智能面临的一项重大挑战。 由于这些障碍，先前方法不得不使用人类专家数据和特定领域的课程学习。 Dreamer 无需任何专门调整，便成为首个从零开始在 Minecraft 中收集到钻石的算法。

2. Learning Algorithm

World Model Learning

Actor Critic Learning

图3：Dreamer 的训练过程。世界模型把感官输入编码为离散表示，并根据动作预测这些表示；行动者与评论家从世界模型预测的抽象轨迹中学习动作和价值。

We present the third generation of the Dreamer algorithm. The algorithm consists of three neural networks: the world model predicts the outcomes of potential actions, the critic judges the value of each outcome, and the actor chooses actions to reach the most valuable outcomes. The components are trained concurrently from replayed experience while the agent interacts with the environment. To succeed across domains, all three components need to accommodate different signal magnitudes and robustly balance terms in their objectives. This is challenging as we are not only targeting similar tasks within the same domain but aim to learn across diverse domains with fixed hyperparameters. This section introduces the world model, critic, and actor along with their robust loss functions, as well as tools for robustly predicting quantities of unknown orders of magnitude.

我们提出第三代 Dreamer 算法。 该算法由三个神经网络组成：世界模型预测潜在动作的结果，评论家判断每种结果的价值，行动者则选择能够达到最有价值结果的动作。 智能体与环境交互时，这些组件会利用经验回放并发训练。 为了跨领域取得成功，三个组件都需要适应不同的信号幅度，并稳健地平衡各自目标函数中的项。 这颇具挑战性，因为我们的目标不仅是在同一领域的相似任务上学习，还要用固定超参数跨不同领域学习。 本节介绍世界模型、评论家和行动者及其稳健损失函数，同时介绍用于稳健预测未知数量级数值的工具。

2.1 World Model Learning

The world model learns compact representations of sensory inputs through autoencoding and enables planning by predicting future representations and rewards for potential actions. We implement the world model as a Recurrent State-Space Model (RSSM), shown in Figure 3. First, an encoder maps sensory inputs $x_t$ to stochastic representations $z_t$. Then, a sequence model with recurrent state $h_t$ predicts the sequence of these representations given past actions $a_{t-1}$. The concatenation of $h_t$ and $z_t$ forms the model state from which we predict rewards $r_t$ and episode continuation flags $c_t\in \{0,1\}$ and reconstruct the inputs to ensure informative representations:

世界模型通过自动编码学习感官输入的紧凑表示，并通过预测潜在动作对应的未来表示和奖励来实现规划。 我们把世界模型实现为 图3 所示的循环状态空间模型（RSSM）。 首先，编码器把感官输入 $x_t$ 映射为随机表示 $z_t$。 然后，具有循环状态 $h_t$ 的序列模型根据过去的动作 $a_{t-1}$ 预测这些表示的序列。 将 $h_t$ 与 $z_t$ 拼接便得到模型状态；我们由此预测奖励 $r_t$ 和回合继续标志 $c_t\in \{0,1\}$，并重建输入以确保表示包含充分信息：

$$\begin{array}{rlrl}{h}_t& =f_{\varphi }(h_{t-1},z_{t-1},a_{t-1})& & \text{Sequence model},\\ z_t& \sim q_{\varphi }(z_t\mid h_t,x_t)& & \text{Encoder},\\ {\hat{z}}_t& \sim p_{\varphi }({\hat{z}}_t\mid h_t)& & \text{Dynamics predictor},\\ {\hat{r}}_t& \sim p_{\varphi }({\hat{r}}_t\mid h_t,z_t)& & \text{Reward predictor},\\ {\hat{c}}_t& \sim p_{\varphi }({\hat{c}}_t\mid h_t,z_t)& & \text{Continue predictor},\\ {\hat{x}}_t& \sim p_{\varphi }({\hat{x}}_t\mid h_t,z_t)& & \text{Decoder}.\end{array}$$

Figure 4 visualizes long-term video predictions of the world world. The encoder and decoder use convolutional neural networks (CNN) for image inputs and multi-layer perceptrons (MLPs) for vector inputs. The dynamics, reward, and continue predictors are also MLPs. The representations are sampled from a vector of softmax distributions and we take straight-through gradients through the sampling step. Given a sequence batch of inputs $x_{1:T}$, actions $a_{1:T}$, rewards $r_{1:T}$, and continuation flags $c_{1:T}$, the world model parameters $\varphi$ are optimized end-to-end to minimize the prediction loss ${\mathcal{L}}_{\mathrm{pred}}$, the dynamics loss ${\mathcal{L}}_{\mathrm{dyn}}$, and the representation loss ${\mathcal{L}}_{\mathrm{rep}}$ with corresponding loss weights ${\beta }_{\mathrm{pred}}=1$, ${\beta }_{\mathrm{dyn}}=1$, and ${\beta }_{\mathrm{rep}}=0.1$:

图4 展示了世界模型的长期视频预测。 对于图像输入，编码器和解码器使用卷积神经网络（CNN）；对于向量输入，则使用多层感知机（MLP）。 动力学、奖励和继续预测器也都是 MLP。 表示从一组 softmax 分布中采样，我们在采样步骤中使用直通梯度。 给定由输入 $x_{1:T}$、动作 $a_{1:T}$、奖励 $r_{1:T}$ 和继续标志 $c_{1:T}$ 构成的序列批次，世界模型参数 $\varphi$ 通过端到端优化，最小化预测损失 ${\mathcal{L}}_{\mathrm{pred}}$、动力学损失 ${\mathcal{L}}_{\mathrm{dyn}}$ 和表示损失 ${\mathcal{L}}_{\mathrm{rep}}$；对应损失权重分别为 ${\beta }_{\mathrm{pred}}=1$、${\beta }_{\mathrm{dyn}}=1$ 和 ${\beta }_{\mathrm{rep}}=0.1$：

$$\mathcal{L}(\varphi )\doteq {\mathbb{E}}_{q_{\varphi }}\left[\sum _{t=1}^T({\beta }_{\mathrm{pred}}{\mathcal{L}}_{\mathrm{pred}}(\varphi )+{\beta }_{\mathrm{dyn}}{\mathcal{L}}_{\mathrm{dyn}}(\varphi )+{\beta }_{\mathrm{rep}}{\mathcal{L}}_{\mathrm{rep}}(\varphi ))\right].$$

图4：DMLab 迷宫（上）和四足机器人（下）的多步视频预测。给定 5 张上下文图像和完整动作序列，模型在无法访问中间图像的情况下预测未来 45 帧，并学习每个环境的底层结构。

The prediction loss trains the decoder and reward predictor via the symlog squared loss described later, and the continue predictor via logistic regression. The dynamics loss trains the sequence model to predict the next representation by minimizing the KL divergence between the predictor $p_{\varphi }(z_t\mid h_t)$ and the next stochastic representation $q_{\varphi }(z_t\mid h_t,x_t)$. The representation loss, in turn, trains the representations to become more predictable allowing us to use a factorized dynamics predictor for fast sampling during imagination training. The two losses differ in the stop-gradient operator $\mathrm{sg}(\cdot )$ and their loss scale. To avoid a degenerate solution where the dynamics are trivial to predict but fail to contain enough information about the inputs, we employ free bits by clipping the dynamics and representation losses below the value of 1 nat $\approx$ 1.44 bits. This disables them while they are already minimized well to focus learning on the prediction loss:

预测损失通过后文所述的 symlog 平方损失训练解码器和奖励预测器，并通过逻辑回归训练继续预测器。 动力学损失通过最小化预测器 $p_{\varphi }(z_t\mid h_t)$ 与下一个随机表示 $q_{\varphi }(z_t\mid h_t,x_t)$ 之间的 KL 散度，训练序列模型预测下一个表示。 表示损失则训练表示使其变得更容易预测，从而让我们能够在想象训练期间使用因子化动力学预测器进行快速采样。 这两个损失的区别在于停止梯度算子 $\mathrm{sg}(\cdot )$ 及其损失尺度。 为了避免动力学虽易于预测却未包含足够输入信息的退化解，我们采用自由比特，把低于 1 nat（$\approx$ 1.44 bits）的动力学损失和表示损失截断。 当它们已经得到充分最小化时，这会将其禁用，使学习专注于预测损失：

$$\begin{array}{rl}{\mathcal{L}}_{\mathrm{pred}}(\varphi )& \doteq -\ln p_{\varphi }(x_t\mid z_t,h_t)-\ln p_{\varphi }(r_t\mid z_t,h_t)-\ln p_{\varphi }(c_t\mid z_t,h_t),\\ {\mathcal{L}}_{\mathrm{dyn}}(\varphi )& \doteq max(1,D_{\mathrm{KL}}[\mathrm{sg}(q_{\varphi }(z_t\mid h_t,x_t))\parallel p_{\varphi }(z_t\mid h_t)]),\\ {\mathcal{L}}_{\mathrm{rep}}(\varphi )& \doteq max(1,D_{\mathrm{KL}}[q_{\varphi }(z_t\mid h_t,x_t)\parallel \mathrm{sg}(p_{\varphi }(z_t\mid h_t))]).\end{array}$$

Previous world models require scaling the representation loss differently based on the visual complexity of the environment. Complex 3D environments contain details unnecessary for control and thus prompt a stronger regularizer to simplify the representations and make them more predictable. In games with static backgrounds and where individual pixels may matter for the task, a weak regularizer is required to extract fine details. We find that combining free bits with a small representation loss resolves this dilemma, allowing for fixed hyperparameters across domains. Moreover, we transform vector observations using the symlog function described later, to prevent large inputs and large reconstruction gradients, further stabilizing the trade-off with the representation loss.

先前的世界模型需要根据环境的视觉复杂度，以不同尺度调整表示损失。 复杂的三维环境包含许多控制所不需要的细节，因此需要更强的正则项来简化表示并使其更容易预测。 而在具有静态背景、单个像素可能影响任务的游戏中，则需要较弱的正则项来提取精细细节。 我们发现，将自由比特与较小的表示损失结合可以解决这一矛盾，从而允许跨领域使用固定超参数。 此外，我们使用后文所述的 symlog 函数变换向量观测，以防止过大的输入和重建梯度，进一步稳定其与表示损失之间的权衡。

We occasionally observed spikes the in KL losses in earlier experiments, consistent with reports for deep variational autoencoders. To prevent this, we parameterize the categorical distributions of the encoder and dynamics predictor as mixtures of 1% uniform and 99% neural network output, making it impossible for them to become deterministic and thus ensuring well-behaved KL losses. Further model details and hyperparameters are included in the supplementary material.

在早期实验中，我们偶尔观察到 KL 损失出现尖峰，这与深度变分自动编码器的相关报告一致。 为了避免这一问题，我们把编码器和动力学预测器的类别分布参数化为 1% 均匀分布与 99% 神经网络输出的混合，使其不可能变为确定性分布，从而确保 KL 损失表现良好。 更多模型细节和超参数见补充材料。

2.2 Critic Learning

The actor and critic neural networks learn behaviors purely from abstract trajectories of representations predicted by the world model. For environment interaction, we select actions by sampling from the actor network without lookahead planning. The actor and critic operate on model states $s_t\doteq \{h_t,z_t\}$ and thus benefit from the Markovian representations learned by the recurrent world model. The actor aims to maximize the return $R_t\doteq \sum _{\tau =0}^{\mathrm{\infty }}{\gamma }^{\tau }{r}_{t+\tau }$ with a discount factor $\gamma =0.997$ for each model state. To consider rewards beyond the prediction horizon $T=16$, the critic learns to approximate the distribution of returns for each state under the current actor behavior:

行动者和评论家神经网络完全从世界模型预测的抽象表示轨迹中学习行为。 与环境交互时，我们通过从行动者网络中采样来选择动作，而不进行前瞻规划。 行动者和评论家在模型状态 $s_t\doteq \{h_t,z_t\}$ 上运行，因此能够受益于循环世界模型学到的马尔可夫表示。 对于每个模型状态，行动者的目标是以折扣因子 $\gamma =0.997$ 最大化回报 $R_t\doteq \sum _{\tau =0}^{\mathrm{\infty }}{\gamma }^{\tau }{r}_{t+\tau }$。 为了考虑超出预测范围 $T=16$ 的奖励，评论家学习近似当前行动者行为下每个状态的回报分布：

$$\begin{array}{rl}\text{Actor:}& a_t\sim {\pi }_{\theta }(a_t\mid s_t),\\ \text{Critic:}& v_{\psi }(R_t\mid s_t).\end{array}$$

Starting from representations of replayed inputs, the world model and actor generate a trajectory of imagined model states $s_{1:T}$, actions $a_{1:T}$, rewards $r_{1:T}$, and continuation flags $c_{1:T}$. Because the critic predicts a distribution, we read out its predicted values $v_t\doteq \mathrm{E}[v_{\psi }(\cdot \mid s_t)]$ as the expectation of the distribution. To estimate returns that consider rewards beyond the prediction horizon, we compute bootstrapped $\lambda$-returns that integrate the predicted rewards and the values. The critic learns to predict the distribution of the return estimates $R_t^{\lambda }$ using the maximum likelihood loss:

从经验回放输入的表示出发，世界模型和行动者生成由想象模型状态 $s_{1:T}$、动作 $a_{1:T}$、奖励 $r_{1:T}$ 和继续标志 $c_{1:T}$ 构成的轨迹。 由于评论家预测的是一个分布，我们把分布的期望作为其预测值 $v_t\doteq \mathrm{E}[v_{\psi }(\cdot \mid s_t)]$ 读出。 为了估计包含预测范围之外奖励的回报，我们计算自举 $\lambda$ 回报，将预测奖励与价值结合起来。 评论家使用最大似然损失学习预测回报估计 $R_t^{\lambda }$ 的分布：

$$\begin{array}{rl}\mathcal{L}(\psi )& \doteq -\sum _{t=1}^T\ln v_{\psi }(R_t^{\lambda }\mid s_t),\\ R_t^{\lambda }& \doteq r_t+\gamma c_t((1-\lambda )v_t+\lambda R_{t+1}^{\lambda }),\\ R_T^{\lambda }& \doteq v_T.\end{array}$$

While a simple choice would be to parameterize the critic as a Normal distribution, the return distribution can have multiple modes and vary by orders of magnitude across environments. To stabilize and accelerate learning under these conditions, we parameterize the critic as categorical distribution with exponentially spaced bins, decoupling the scale of gradients from the prediction targets as described later. To improve value prediction in environments where rewards are challenging to predict, we apply the critic loss both to imagined trajectories with loss scale ${\beta }_{\mathrm{val}}=1$ and to trajectories sampled from the replay buffer with loss scale ${\beta }_{\mathrm{repval}}=0.3$. The critic replay loss uses the imagination returns $R_t^{\lambda }$ at the start states of the imagination rollouts as on-policy value annotations for the replay trajectory to then compute $\lambda$-returns over the replay rewards.

一种简单的做法是把评论家参数化为正态分布，但回报分布可能具有多个模态，并且在不同环境之间相差多个数量级。 为了在这些条件下稳定并加速学习，我们把评论家参数化为具有指数间隔分箱的类别分布，从而按照后文所述方式将梯度尺度与预测目标解耦。 为了在奖励难以预测的环境中改进价值预测，我们既把评论家损失应用于损失尺度为 ${\beta }_{\mathrm{val}}=1$ 的想象轨迹，也应用于从经验回放缓冲区采样、损失尺度为 ${\beta }_{\mathrm{repval}}=0.3$ 的轨迹。 评论家回放损失把想象展开起始状态处的想象回报 $R_t^{\lambda }$ 用作回放轨迹的在策略价值标注，再根据回放奖励计算 $\lambda$ 回报。

Because the critic regresses targets that depend on its own predictions, we stabilize learning by regularizing the critic towards predicting the outputs of an exponentially moving average of its own parameters. This is similar to target networks used previously in reinforcement learning but allows us to compute returns using the current critic network. We further noticed that the randomly initialized reward predictor and critic networks at the start of training can result in large predicted rewards that can delay the onset of learning. We thus initialize the output weight matrix of the reward predictor and critic to zeros, which alleviates the problem and accelerates early learning.

由于评论家回归的目标依赖其自身预测，我们通过正则化评论家使其趋向于预测自身参数指数移动平均的输出，从而稳定学习。 这类似于先前强化学习中使用的目标网络，但允许我们使用当前评论家网络来计算回报。 我们还注意到，在训练开始时随机初始化的奖励预测器和评论家网络可能产生较大的预测奖励，从而延迟学习启动。 因此，我们把奖励预测器和评论家的输出权重矩阵初始化为零，这缓解了该问题并加速早期学习。

2.3 Actor Learning

The actor learns to choose actions that maximize return while exploring through an entropy regularizer. However, the correct scale for this regularizer depends both on the scale and frequency of rewards in the environment. Ideally, we would like the agent to explore more if rewards are sparse and exploit more if rewards are dense or nearby. At the same time, the exploration amount should not be influenced by arbitrary scaling of rewards in the environment. This requires normalizing the return scale while preserving information about reward frequency.

行动者学习选择能够最大化回报的动作，同时通过熵正则项进行探索。 然而，该正则项的正确尺度同时取决于环境中奖励的尺度和频率。 理想情况下，当奖励稀疏时，我们希望智能体进行更多探索；当奖励密集或近在眼前时，则进行更多利用。 同时，探索量不应受到环境中奖励任意缩放的影响。 这要求我们在保留奖励频率信息的同时，对回报尺度进行归一化。

To use a fixed entropy scale of $\eta =3\times {10}^{-4}$ across domains, we normalize returns to be approximately contained in the interval $[0,1]$. In practice, substracting an offset from the returns does not change the actor gradient and thus dividing by the range $S$ is sufficient. Moreover, to avoid amplifying noise from function approximation under sparse rewards, we only scale down large return magnitudes but leave small returns below the threshold of $L=1$ untouched. We use the Reinforce estimator for both discrete and continuous actions, resulting in the surrogate loss function:

为了跨领域使用固定熵尺度 $\eta =3\times {10}^{-4}$，我们对回报进行归一化，使其大致位于区间 $[0,1]$ 内。 在实践中，从回报中减去偏移量不会改变行动者梯度，因此除以范围 $S$ 就足够了。 此外，为避免在稀疏奖励下放大函数近似产生的噪声，我们只缩小幅度较大的回报，而保持低于阈值 $L=1$ 的小回报不变。 我们对离散动作和连续动作都使用 Reinforce 估计器，从而得到以下替代损失函数：

$$\mathcal{L}(\theta )\doteq -\sum _{t=1}^T[\mathrm{sg}\left(\frac{R_t^{\lambda }-v_{\psi }(s_t)}{max(1,S)}\right)\log {\pi }_{\theta }(a_t\mid s_t)+\eta \mathcal{H}[{\pi }_{\theta }(a_t\mid s_t)]].$$

The return distribution can be multi-modal and include outliers, especially for randomized environments where some episodes have higher achievable returns than others. Normalizing by the smallest and largest observed returns would then scale returns down too much and may cause suboptimal convergence. To be robust to these outliers, we compute the range from the 5^th to the 95^th return percentile over the return batch and smooth out the estimate using an exponential moving average:

回报分布可能是多模态的并包含离群值，尤其是在随机化环境中，某些回合能够达到的回报高于其他回合。 此时，使用观测到的最小和最大回报进行归一化会把回报缩小得过多，并可能导致次优收敛。 为了稳健应对这些离群值，我们在回报批次上计算第 5 至第 95 百分位数之间的范围，并使用指数移动平均平滑该估计：

$$S\doteq \mathrm{EMA}(\mathrm{Per}(R_t^{\lambda },95)-\mathrm{Per}(R_t^{\lambda },5),0.99).$$

Previous work typically normalizes advantages rather than returns, which puts a fixed amount of emphasis on maximizing returns over entropy regardless of whether rewards are within reach. Scaling up advantages when rewards are sparse can amplify noise that outweighs the entropy regularizer and stagnates exploration. Normalizing rewards or returns by standard deviation can fail under sparse rewards where their standard deviation is near zero, drastically amplifying rewards regardless of their size. Constrained optimization targets a fixed entropy on average across states regardless of achievable returns, which is robust but explores slowly under sparse rewards and converges lower under dense rewards. We did not find stable hyperparameters across domains for these approaches. Return normalization with a denominator limit overcomes these challenges, exploring rapidly under sparse rewards and converging to high performance across diverse domains.

先前工作通常对优势而非回报进行归一化；无论奖励是否触手可及，这都会固定最大化回报相对于熵的强调程度。 在奖励稀疏时放大优势可能会放大噪声，使其超过熵正则项并令探索停滞。 使用标准差归一化奖励或回报，在奖励稀疏、标准差接近于零时可能失效，无论奖励大小如何都会将其大幅放大。 约束优化以跨状态平均固定的熵为目标，而不考虑可达到的回报；这种方法较为稳健，但在稀疏奖励下探索缓慢，在密集奖励下收敛性能较低。 我们未能为这些方法找到跨领域稳定的超参数。 带分母下限的回报归一化克服了这些挑战，能够在稀疏奖励下快速探索，并在多样化领域中收敛到高性能。

2.4 Robust Predictions

Reconstructing inputs and predicting rewards and returns can be challenging because the scale of these quantities can vary across domains. Predicting large targets using a squared loss can lead to divergence whereas absolute and Huber losses stagnate learning. On the other hand, normalizing targets based on running statistics introduces non-stationarity into the optimization. We suggest the symlog squared error as a simple solution to this dilemma. For this, a neural network $f(x,\theta )$ with inputs $x$ and parameters $\theta$ learns to predict a transformed version of its targets $y$. To read out predictions $\hat{y}$ of the network, we apply the inverse transformation:

由于这些量的尺度在不同领域之间可能不同，重建输入以及预测奖励和回报可能颇具挑战性。 使用平方损失预测较大的目标可能导致发散，而绝对损失和 Huber 损失则会使学习停滞。 另一方面，根据运行统计量对目标进行归一化，会给优化过程引入非平稳性。 我们建议使用 symlog 平方误差作为解决这一困境的简单方法。 为此，输入为 $x$、参数为 $\theta$ 的神经网络 $f(x,\theta )$ 学习预测目标 $y$ 的变换版本。 为了读出网络的预测 $\hat{y}$，我们应用逆变换：

$$\begin{array}{rl}\mathcal{L}(\theta )& \doteq \frac{1}{2}(f(x,\theta )-\mathrm{symlog}(y){)}^2,\\ \hat{y}& \doteq \mathrm{symexp}(f(x,\theta )).\end{array}$$

Using the logarithm as transformation would not allow us to predict targets that take on negative values. Therefore, we choose a function from the bi-symmetric logarithmic family that we name symlog as the transformation with the symexp function as its inverse:

使用对数作为变换将使我们无法预测取负值的目标。 因此，我们从双对称对数函数族中选择一个函数，将其命名为 symlog，并以 symexp 函数作为其逆函数：

$$\begin{array}{rl}\mathrm{symlog}(x)& \doteq \mathrm{sign}(x)\ln (|x|+1),\\ \mathrm{symexp}(x)& \doteq \mathrm{sign}(x)(\exp (|x|)-1).\end{array}$$

The symlog function compresses the magnitudes of both large positive and negative values. Unlike the logarithm, it is symmetric around the origin while preserving the input sign. This allows the optimization process to quickly move the network predictions to large values when needed. The symlog function approximates the identity around the origin so that it does not affect learning of targets that are already small enough.

symlog 函数会压缩较大正值和负值的幅度。 与对数不同，它在保留输入符号的同时关于原点对称。 这使优化过程能够在需要时迅速把网络预测移动到较大的数值。 symlog 函数在原点附近近似恒等映射，因此不会影响对已经足够小的目标的学习。

For potentially stochastic targets, such as rewards or returns, we introduce the symexp twohot loss. Here, the network outputs the logits for a softmax distribution over exponentially spaced bins $b_i\in B$. Predictions are read out as the weighted average of the bin positions weighted by their predicted probabilities. Importantly, the network can output any continuous value in the interval because the weighted average can fall between the buckets:

对于奖励或回报等可能具有随机性的目标，我们引入 symexp twohot 损失。 在这里，网络输出定义在指数间隔分箱 $b_i\in B$ 上的 softmax 分布 logits。 预测值通过分箱位置的加权平均读出，权重为各分箱的预测概率。 重要的是，由于加权平均可以落在两个分箱之间，网络能够输出该区间内的任意连续值：

$$\begin{array}{rl}\hat{y}& \doteq \mathrm{softmax}(f(x){)}^{\mathsf{T}}B,\\ B& \doteq \mathrm{symexp}\left(\left[\begin{array}{ccc}-20& \dots & +20\end{array}\right]\right).\end{array}$$

The network is trained on twohot encoded targets, a generalization of onehot encoding to continuous values. The twohot encoding of a scalar is a vector with $|B|$ entries that are all $0$ except at the indices $k$ and $k+1$ of the two bins closest to the encoded scalar. The two entries sum up to $1$, with linearly higher weight given to the bin that is closer to the encoded continuous number. The network is then trained to minimize the categorical cross entropy loss for classification with soft targets. Note that the loss only depends on the probabilities assigned to the bins but not on the continuous values associated with the bin locations, decoupling the size of the gradients from the size of the targets:

网络使用 twohot 编码目标进行训练，这是 onehot 编码向连续值的推广。 标量的 twohot 编码是一个含有 $|B|$ 个元素的向量；除与被编码标量最接近的两个分箱所对应的索引 $k$ 和 $k+1$ 外，其余元素均为 $0$。 这两个元素之和为 $1$，其中距离被编码连续数值更近的分箱以线性方式获得更高权重。 随后，网络通过最小化带软目标分类的类别交叉熵损失进行训练。 请注意，该损失只依赖于分配给分箱的概率，而不依赖于与分箱位置关联的连续值，从而将梯度大小与目标大小解耦：

$$\mathcal{L}(\theta )\doteq -\mathrm{twohot}(y{)}^{\mathsf{T}}\log \mathrm{softmax}(f(x,\theta )).$$

Applying these principles, Dreamer transforms vector observations using the symlog functions, both for the encoder inputs and the decoder targets and employs the synexp twohot loss for the reward predictor and critic. We find that these techniques enable robust and fast learning across many diverse domains. For critic learning, an alternative asymmetric transformation has previously been proposed, which we found less effective on average across domains. Unlike alternatives, symlog transformations avoid truncating large targets, introducing non-stationary from normalization, or adjusting network weights when new extreme values are detected.

应用这些原则后，Dreamer 使用 symlog 函数变换向量观测，并将其同时用于编码器输入和解码器目标；奖励预测器和评论家则采用 synexp twohot 损失。 我们发现，这些技术能够在许多不同领域中实现稳健且快速的学习。 先前研究曾为评论家学习提出另一种非对称变换，但我们发现它在各领域的平均效果较差。 与其他方法不同，symlog 变换能够避免截断较大目标、因归一化引入非平稳性，或在检测到新的极值时调整网络权重。

3. Results

We evaluate the generality of Dreamer across 8 domains---with over 150 tasks---under fixed hyperparameters. We designed the experiments to compare Dreamer to the best methods in the literature, which are often specifically designed and tuned for the benchmark at hand. We further compare to a high-quality implementation of PPO, a standard reinforcement learning algorithm that is known for its robustness. We run PPO with fixed hyperparameters chosen to maximize performance across domains and that reproduce strong published results of PPO on ProcGen. To push the boundaries of reinforcement learning, we apply Dreamer to the challenging video game Minecraft, comparing it to strong previous algorithms. Finally, we analyze the importance of individual components of Dreamer and its robustness to different model sizes and computational budgets. All Dreamer agents are trained on a single Nvidia A100 GPU each, making it reproducible for many research labs. A public implementation of Dreamer that reproduces all results is available on the project website.

我们在固定超参数下，跨 8 个领域的 150 多项任务评估 Dreamer 的通用性。 我们设计这些实验，是为了将 Dreamer 与文献中的最佳方法比较；这些方法通常针对当前基准进行专门设计和调优。 我们还将其与 PPO 的高质量实现进行比较；PPO 是一种以稳健性著称的标准强化学习算法。 我们使用固定超参数运行 PPO；这些超参数旨在最大化跨领域性能，并能复现 PPO 在 ProcGen 上已发表的强劲结果。 为了拓展强化学习的能力边界，我们把 Dreamer 应用于具有挑战性的电子游戏 Minecraft，并将其与先前的强大算法进行比较。 最后，我们分析 Dreamer 各个组件的重要性，以及它对不同模型规模和计算预算的稳健性。 每个 Dreamer 智能体都在一张 Nvidia A100 GPU 上训练，因此许多研究实验室都能够复现。 能够复现所有结果的 Dreamer 公开实现可在项目网站上获取。

Benchmarks. We perform an extensive empirical study across 8 domains that include continuous and discrete actions, visual and low-dimensional inputs, dense and sparse rewards, different reward scales, 2D and 3D worlds, and procedural generation. Figure 1 summarizes the benchmark results, showing that Dreamer outperforms a wide range of previous expert algorithms across diverse domains. Crucially, Dreamer substantially outperforms PPO across all domains.

基准。 我们在 8 个领域中开展了广泛的实证研究，这些领域涵盖连续与离散动作、视觉与低维输入、密集与稀疏奖励、不同奖励尺度、二维与三维世界以及程序生成。 图1 汇总了基准结果，表明 Dreamer 在多样化领域中超越了大量先前的专家算法。 关键的是，Dreamer 在所有领域都显著优于 PPO。

• Atari. This established benchmark contains 57 Atari 2600 games with a budget of 200M frames, posing a diverse range of challenges. We use the sticky action simulator setting. Dreamer outperforms the powerful MuZero algorithm while using only a fraction of the computational resources. Dreamer also outperforms the widely-used expert algorithms Rainbow and IQN.

• ProcGen. This benchmark of 16 games features randomized levels and visual distractions to test the robustness and generalization of agents. Within the budget of 50M frames, Dreamer matches the tuned expert algorithm PPG and outperforms Rainbow. Our PPO agent with fixed hyperparameters matches the published score of the highly tuned official PPO implementation.

• DMLab. This suite of 30 tasks features 3D environments that test spatial and temporal reasoning. In 100M frames, Dreamer exceeds the performance of the scalable IMPALA and R2D2+ agents at 1B environment steps, amounting to a data-efficiency gain of over 1000%. We note that these baselines were not designed for data-efficiency but serve as a valuable comparison point for the performance previously achievable at scale.

• Atari100k. This data-efficiency benchmark comntains 26 Atari games and a budget of only 400K frames, amounting to 2 hours of game time. EfficientZero holds the state-of-the-art by combining online tree search, prioritized replay, and hyperparameter scheduling, but also resets levels early to increase data diversity, making a comparison difficult. Without this complexity, Dreamer outperforms the best remaining methods, including the transformer-based IRIS and TWM agents, the model-free SPR, and SimPLe.

• Proprio Control. This benchmark contains 18 control tasks with continuous actions, proprioceptive vector inputs, and a budget of 500K environment steps. The tasks range from classical control over locomotion to robot manipulation tasks, featuring dense and sparse rewards. Dreamer sets a new state-of-the-art on this benchmark, outperforming D4PG, DMPO, and MPO.

• Visual Control. This benchmark consists of 20 continuous control tasks where the agent receives only high-dimensional images as input and has a budget of 1M environment steps. Dreamer establishes a new state-of-the-art on this benchmark, outperforming DrQ-v2 and CURL, which are specialized to visual environments and leverage data augmentation.

• BSuite. This benchmark includes 23 environments with a total of 468 configurations that are specifically designed to test credit assignment, robustness to reward scale and stochasticity, memory, generalization, and exploration. Dreamer establishes a new state-of-the-art on this benchmark, outperforming Boot DQN and other methods. Dreamer improves over previous algorithms especially in the scale robustness category.

• Atari。 这一经典基准包含 57 款 Atari 2600 游戏，预算为 2 亿帧，提出了多种不同挑战。我们采用粘滞动作模拟器设置。Dreamer 只使用一小部分计算资源便超越了强大的 MuZero 算法。Dreamer 还超越了广泛使用的专家算法 Rainbow 和 IQN。

• ProcGen。 这一包含 16 款游戏的基准通过随机化关卡和视觉干扰来测试智能体的稳健性与泛化能力。在 5000 万帧预算内，Dreamer 达到经过调优的专家算法 PPG 的水平，并超越 Rainbow。采用固定超参数的 PPO 智能体取得了与高度调优的官方 PPO 实现所发表分数相当的结果。

• DMLab。 这套包含 30 项任务的基准采用测试空间与时间推理能力的三维环境。在 1 亿帧下，Dreamer 超过了可扩展的 IMPALA 和 R2D2+ 智能体在 10 亿环境步时的性能，数据效率提升超过 1000%。我们指出，这些基线并非为数据效率而设计，但它们为此前大规模计算下可达到的性能提供了有价值的比较点。

• Atari100k。 这一数据效率基准包含 26 款 Atari 游戏，预算只有 40 万帧，相当于 2 小时游戏时间。EfficientZero 通过结合在线树搜索、优先经验回放和超参数调度保持当前最佳性能，但它也会提前重置关卡以增加数据多样性，因此难以直接比较。无需这些复杂机制，Dreamer 便超越了其余最佳方法，包括基于 Transformer 的 IRIS 和 TWM 智能体、无模型的 SPR 以及 SimPLe。

• 本体感知控制。 该基准包含 18 项控制任务，具有连续动作、本体感知向量输入以及 50 万环境步预算。任务从经典控制、运动控制一直延伸到机器人操作，并同时包含密集奖励和稀疏奖励。Dreamer 在这一基准上建立了新的最佳性能，超越 D4PG、DMPO 和 MPO。

• 视觉控制。 该基准包含 20 项连续控制任务，智能体只接收高维图像作为输入，预算为 100 万环境步。Dreamer 在该基准上建立了新的最佳性能，超过专门面向视觉环境并利用数据增强的 DrQ-v2 和 CURL。

• BSuite。 该基准包含 23 个环境、共 468 种配置，专门用于测试信用分配、对奖励尺度与随机性的稳健性、记忆、泛化和探索能力。Dreamer 在这一基准上建立了新的最佳性能，超越 Boot DQN 等方法。Dreamer 尤其在尺度稳健性类别中优于先前算法。

Minecraft. Collecting diamonds in the popular game Minecraft has been a long-standing challenge in artificial intelligence. Every episode in this game is set in a unique randomly generated and infinite 3D world. Episodes last until the player dies or up to 36000 steps equaling 30 minutes, during which the player needs to discover a sequence of 12 items from sparse rewards by foraging for resources and crafting tools. It takes about 20 minutes for experienced human players to obtain diamonds. We follow the block breaking setting of prior work because the provided action space would make it challenging for stochastic policies to keep a key pressed for a prolonged time.

Minecraft。 在流行游戏 Minecraft 中收集钻石，长期以来一直是人工智能面临的一项挑战。 该游戏的每个回合都位于一个独特、随机生成且无限的三维世界中。 回合会持续到玩家死亡，或最多运行 36000 步、相当于 30 分钟；在此期间，玩家需要搜寻资源并制作工具，从稀疏奖励中发现一个包含 12 种物品的序列。 即使是经验丰富的人类玩家，也需要约 20 分钟才能获得钻石。 我们沿用先前工作的方块破坏设置，因为所提供的动作空间会使随机策略难以长时间持续按住一个按键。

图5：Minecraft Diamond 任务中，训练智能体发现最后三种物品的比例。先前算法最多推进到铁镐，Dreamer 是唯一能够发现钻石且稳定做到这一点的对比算法。

Because of the training time in this complex domain, extensive tuning would be difficult for Minecraft. Instead, we apply Dreamer out of the box with its default hyperparameters. As shown in Figure 5 and Figure 1, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without using human data as was required by VPT or adaptive curricula. All the Dreamer agents we trained on Minecraft discover diamonds in 100M environment steps. While several strong baselines progress to advanced items such as the iron pickaxe, none of them discovers a diamond.

由于这个复杂领域的训练耗时很长，针对 Minecraft 进行广泛调优并不现实。 因此，我们直接使用 Dreamer 的默认超参数，不做额外调整。 如 图5 和 图1 所示，Dreamer 是首个无需 VPT 所依赖的人类数据或自适应课程，便能从零开始在 Minecraft 中收集钻石的算法。 我们在 Minecraft 上训练的所有 Dreamer 智能体，都能在 1 亿环境步内发现钻石。 尽管多个强基线能够推进到铁镐等高级物品，但它们都未能发现钻石。

Ablations. In Figure 6, we ablate the robustness techniques and learning signals on a diverse set of 14 tasks to understand their importance. The training curves of individual tasks are included in the supplementary material. We observe that all robustness techniques contribute to performance, most notably the KL objective of the world model, followed by return normalization and symexp twohot regression for reward and value prediction. In general, we find that each individual technique is critical on a subset of tasks but may not affect performance on other tasks.

消融。 在 图6 中，我们在一组多样化的 14 项任务上消融稳健性技术和学习信号，以了解它们的重要性。 各项任务的训练曲线收录在补充材料中。 我们观察到，所有稳健性技术都有助于提升性能，其中最重要的是世界模型的 KL 目标，其次是回报归一化，以及用于奖励和价值预测的 symexp twohot 回归。 总体而言，我们发现每项技术都对部分任务至关重要，但可能不影响其他任务的性能。

图6：Dreamer 的消融与稳健扩展。a：各项稳健性技术平均而言都有贡献。b：Dreamer 的性能主要依赖世界模型的无监督重建损失。c：模型从 1200 万扩展到 4 亿参数时，性能单调提升且所需环境交互减少。d：更高的经验回放比能够可预测地提升性能。

To investigate the effect of the world model, we ablate the learning signals of Dreamer by stopping either the task-specific reward and value prediction gradients or the task-agnostic reconstruction gradients from shaping its representations. Unlike previous reinforcement learning algorithms that often rely only on task-specific learning signals, Dreamer rests predominantly on the unsupervised objective of its world model. This finding could allow for future algorithm variants that leverage pretraining on unsupervised data.

为了研究世界模型的作用，我们分别阻止特定任务的奖励与价值预测梯度，或与任务无关的重建梯度塑造 Dreamer 的表示，从而消融其学习信号。 先前的强化学习算法往往只依赖特定任务的学习信号；与之不同，Dreamer 主要依靠其世界模型的无监督目标。 这一发现可能催生利用无监督数据预训练的未来算法变体。

Scaling properties. To investigate whether Dreamer can scale robustly, we train 6 model sizes ranging from 12M to 400M parameters, as well as different replay ratios on Crafter and a DMLab task. The replay ratio affects the number of gradient updates performed by the agent. Figure 6 shows robust learning with fixed hyperparameters across the compared model sizes and replay ratios. Moreover, increasing the model size directly translates to both higher task performance and a lower data requirement. Increasing the number of gradient steps further reduces the interactions needed to learn successful behaviors. The results show that Dreamer learns robustly across model sizes and replay ratios and that its performance and provides a predictable way for increasing performance given computational resources.

扩展特性。 为了研究 Dreamer 能否稳健扩展，我们在 Crafter 和一项 DMLab 任务上训练了从 1200 万到 4 亿参数的 6 种模型规模，并采用不同的经验回放比。 经验回放比会影响智能体执行的梯度更新次数。 图6 表明，在所比较的不同模型规模和经验回放比下，固定超参数仍能实现稳健学习。 此外，增大模型规模会直接带来更高的任务性能和更低的数据需求。 增加梯度更新步数还会进一步减少学习成功行为所需的交互次数。 结果表明，Dreamer 能够跨模型规模和经验回放比稳健学习，并提供一种在给定计算资源下可预测地提升性能的方式。

4. Previous Work

Developing general-purpose algorithms has long been a goal of reinforcement learning research. PPO is one of the most widely used algorithms and is relatively robust but requires large amounts of experience and often yields lower performance than specialized alternatives. SAC is a popular choice for continuous control and leverages experience replay for data-efficiency, but in practice requires tuning, especially for its entropy scale, and struggles under high-dimensional inputs. MuZero plans using a value prediction model and has been applied to board games and Atari, but the authors did not release an implementation and the algorithm contains several complex components, making it challenging to reproduce. Gato fits one large model to expert demonstrations of multiple tasks, but is only applicable when expert data is available. In comparison, Dreamer masters a diverse range of environments with fixed hyperparameters, does not require expert data, and its implementation is open source.

开发通用算法长期以来一直是强化学习研究的目标。 PPO 是使用最广泛的算法之一，也相对稳健，但它需要大量经验，而且性能通常低于专用替代方法。 SAC 是连续控制中的常用选择，并利用经验回放提高数据效率；但在实践中，它需要调优，尤其是熵尺度，并且难以处理高维输入。 MuZero 使用价值预测模型进行规划，已被应用于棋盘游戏和 Atari；但其实现并未公开，而且算法包含多个复杂组件，因此难以复现。 Gato 使用一个大型模型拟合多项任务的专家示范，但只适用于能够获得专家数据的情况。 相比之下，Dreamer 使用固定超参数便能掌握多种不同环境，不需要专家数据，并且实现已经开源。

Minecraft has been a focus of recent research. With MALMO, Microsoft released a free version of the successful game for research purposes. MineRL offers several competition environments, which we rely on as the basis for our experiments. The MineRL competition supports agents in exploring and learning meaningful skills through a diverse human dataset. Voyager obtains items at a similar depth in the technology tree as Dreamer using API calls to a language model but operates on top of the MineFlayer bot scripting layer that was specifically engineered to the game and exposes high-level actions. VPT trained an agent to play Minecraft through behavioral cloning based on expert data of keyboard and mouse actions collected by contractors and finetuning using reinforcement learning to obtain diamonds using 720 GPUs for 9 days. In comparison, Dreamer uses the MineRL competition action space to autonomously learn to collect diamonds from sparse rewards using 1 GPU for 9 days, without human data.

Minecraft 一直是近期研究的焦点。 Microsoft 通过 MALMO 发布了这款成功游戏的免费版本，用于研究目的。 MineRL 提供了多个竞赛环境，我们以此作为实验基础。 MineRL 竞赛通过多样化的人类数据集，帮助智能体探索并学习有意义的技能。 Voyager 通过调用语言模型 API，能够获得与 Dreamer 在技术树上深度相近的物品；但它运行在专门为该游戏设计并暴露高级动作的 MineFlayer 机器人脚本层之上。 VPT 使用承包人员采集的键盘和鼠标操作专家数据，通过行为克隆训练智能体玩 Minecraft，再用强化学习微调以获得钻石；整个过程使用 720 张 GPU 训练了 9 天。 相比之下，Dreamer 使用 MineRL 竞赛动作空间，在没有人类数据的情况下，仅用 1 张 GPU 训练 9 天，便能从稀疏奖励中自主学会收集钻石。

5. Conclusion

We present the third generation of the Dreamer algorithm, a general reinforcement learning algorithm that masters a wide range of domains with fixed hyperparameters. Dreamer excels not only across over 150 tasks but also learns robustly across varying data and compute budgets, moving reinforcement learning toward a wide range of practical applications. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch, achieving a significant milestone in the field of artificial intelligence. As a high-performing algorithm that is based on a learned world model, Dreamer paves the way for future research directions, including teaching agents world knowledge from internet videos and learning a single world model across domains to allow artificial agents to build up increasingly general knowledge and competency.

我们提出第三代 Dreamer 算法，这是一种使用固定超参数便能掌握广泛领域的通用强化学习算法。 Dreamer 不仅在 150 多项任务上表现出色，还能跨不同数据与计算预算稳健学习，从而推动强化学习走向广泛的实际应用。 Dreamer 无需任何专门调整，便成为首个从零开始在 Minecraft 中收集到钻石的算法，取得了人工智能领域的一项重要里程碑。 Dreamer 是一种建立在已学习世界模型之上的高性能算法，为未来研究方向铺平了道路，其中包括通过互联网视频向智能体传授世界知识，以及跨领域学习单一世界模型，使人工智能体能够积累日益通用的知识与能力。