---

Offline Reinforcement Learning with Implicit Q-Learning

Kostrikov I, Nair A, Levine S. Offline Reinforcement Learning with Implicit Q-Learning. ICLR, 2022.

https://openreview.net/forum?id=68n2s9ZJWF8

https://github.com/ikostrikov/implicit_q_learning

RLQ-LearningICLR 2022CCF-A加州大学伯克利分校

---

基于隐式 Q 学习的离线强化学习

Abstract

Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose a new offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization.

离线强化学习需要协调两个相互冲突的目标：学习一个优于收集数据集的行为策略的策略，同时尽量减小其与行为策略的偏差，以避免分布偏移造成的误差。 这种权衡至关重要，因为当前大多数离线强化学习方法都需要在训练期间查询未见动作的价值以改进策略，因此必须约束这些动作处于分布内，或者对其价值进行正则化。 我们提出一种新的离线强化学习方法，它从不需要评估数据集之外的动作，却仍能通过泛化使学习到的策略显著优于数据中的最佳行为。

The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action.

我们的主要见解是，不再评估最新策略产生的未见动作，而是将状态价值函数视为一个随机变量，其随机性由动作决定，同时仍对环境动态进行积分以避免过度乐观；随后取该随机变量的状态条件上侧期望分位数，以估计该状态下最佳动作的价值，从而隐式逼近策略改进步骤。 这利用了函数逼近器的泛化能力，可以估计给定状态下最佳可用动作的价值，而无需使用这个未见动作直接查询 Q 函数。

Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function, without any explicit policy. Then, we extract the policy via advantage-weighted behavioral cloning, which also avoids querying out-of-sample actions. We dub our method implicit Q-learning (IQL). IQL is easy to implement, computationally efficient, and only requires fitting an additional critic with an asymmetric L2 loss. IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

我们的算法在拟合这种上侧期望分位价值函数与将其回传到 Q 函数之间交替进行，不需要任何显式策略。 随后，我们通过优势加权行为克隆提取策略，这同样避免查询样本外动作。 我们将该方法称为隐式 Q 学习（implicit Q-learning，IQL）。 IQL 易于实现、计算高效，并且只需额外拟合一个采用非对称 L2 损失的评论器。 IQL 在离线强化学习的标准基准 D4RL 上取得了最先进的性能。 我们还表明，IQL 在离线初始化后利用在线交互进行微调时也能取得强劲表现。

1. Introduction

Offline reinforcement learning (RL) addresses the problem of learning effective policies entirely from previously collected data, without online interaction. This is very appealing in a range of real-world domains, from robotics to logistics and operations research, where real-world exploration with untrained policies is costly or dangerous, but prior data is available. However, this also carries with it major challenges: improving the policy beyond the level of the behavior policy that collected the data requires estimating values for actions other than those that were seen in the dataset, and this, in turn, requires trading off policy improvement against distributional shift, since the values of actions that are too different from those in the data are unlikely to be estimated accurately.

离线强化学习（RL）研究完全依靠先前收集的数据、而不进行在线交互来学习有效策略的问题。 这对机器人、物流和运筹学等众多现实领域很有吸引力，因为使用未经训练的策略在现实世界中探索成本高昂或十分危险，而已有数据可供使用。 然而，这也带来了重大挑战：要将策略改进到超过收集数据的行为策略，就需要估计数据集中未见动作的价值；这又要求在策略改进与分布偏移之间进行权衡，因为与数据中的动作差异过大的动作，其价值不太可能得到准确估计。

Prior methods generally address this by either constraining the policy to limit how far it deviates from the behavior policy, or by regularizing the learned value functions to assign low values to out-of-distribution actions. Nevertheless, this imposes a trade-off between how much the policy improves and how vulnerable it is to misestimation due to distributional shift. Can we devise an offline RL method that avoids this issue by never needing to directly query or estimate values for actions that were not seen in the data?

以往方法通常通过约束策略以限制其偏离行为策略的程度，或者对学习到的价值函数进行正则化、使其为分布外动作赋予较低价值来解决这一问题。 然而，这会在策略改进幅度与其受分布偏移所致误估影响的程度之间形成权衡。 我们能否设计一种离线强化学习方法，使其永远不需要直接查询或估计数据中未见动作的价值，从而避免这一问题？

In this work, we start from an observation that in-distribution constraints widely used in prior work might not be sufficient to avoid value function extrapolation, and we ask whether it is possible to learn an optimal policy with in-sample learning, without ever querying the values of any unseen actions. The key idea in our method is to approximate an upper expectile of the distribution over values with respect to the distribution of dataset actions for each state. We alternate between fitting this value function with expectile regression, and then using it to compute Bellman backups for training the $Q$-function. We show that we can do this simply by modifying the loss function in a SARSA-style TD backup, without ever using out-of-sample actions in the target value.

在本文中，我们首先观察到，以往工作广泛使用的分布内约束可能不足以避免价值函数外推，并进一步探究能否通过样本内学习来学习最优策略，而永远不查询任何未见动作的价值。 我们方法的关键思想是，对每个状态，依据数据集动作分布来逼近价值分布的一个上侧期望分位数。 我们交替使用期望分位回归拟合这个价值函数，再利用它计算用于训练 $Q$ 函数的贝尔曼回传。 我们表明，只需修改 SARSA 风格时序差分回传中的损失函数即可做到这一点，并且目标价值中始终不使用样本外动作。

Once this $Q$-function has converged, we extract the corresponding policy using advantage-weighted behavioral cloning. This approach does not require explicit constraints or explicit regularization of out-of-distribution actions during value function training, though our policy extraction step does implicitly enforce a constraint, as discussed in prior work on advantage-weighted regression.

当这个 $Q$ 函数收敛后，我们使用优势加权行为克隆提取相应策略。 这种方法在价值函数训练期间无需对分布外动作施加显式约束或显式正则化，不过正如优势加权回归的既有工作所讨论的那样，我们的策略提取步骤会隐式施加约束。

Our main contribution is implicit Q-learning (IQL), a new offline RL algorithm that avoids ever querying values of unseen actions while still being able to perform multi-step dynamic programming updates. Our method is easy to implement by making a small change to the loss function in a simple SARSA-like TD update and is computationally very efficient. Furthermore, our approach demonstrates the state-of-the-art performance on D4RL, a popular benchmark for offline reinforcement learning. In particular, our approach significantly improves over the prior state-of-the-art on challenging Ant Maze tasks that require to “stitch” several sub-optimal trajectories. Finally, we demonstrate that our approach is suitable for finetuning; after initialization from offline RL, IQL is capable of improving policy performance utilizing additional interactions.

我们的主要贡献是隐式 Q 学习（IQL），这是一种新的离线强化学习算法；它从不查询未见动作的价值，同时仍能执行多步动态规划更新。 我们的方法只需对一个简单的类 SARSA 时序差分更新中的损失函数做很小改动，易于实现且计算效率很高。 此外，我们的方法在流行的离线强化学习基准 D4RL 上取得了最先进的性能。 特别是在需要“拼接”多条次优轨迹的高难度 Ant Maze 任务上，我们的方法显著超过此前最先进的方法。 最后，我们证明了该方法适合微调；经过离线强化学习初始化后，IQL 能够利用额外交互进一步提升策略性能。

2. Related Work

A significant portion of recently proposed offline RL methods are based on either constrained or regularized approximate dynamic programming (e.g., Q-learning or actor-critic methods), with the constraint or regularizer serving to limit deviation from the behavior policy. We will refer to these methods as “multi-step dynamic programming” algorithms, since they perform true dynamic programming for multiple iterations, and therefore can in principle recover the optimal policy if provided with high-coverage data. The constraints can be implemented via an explicit density model, implicit divergence constraints, or by adding a supervised learning term to the policy improvement objective. Several works have also proposed to directly regularize the Q-function to produce low values for out-of-distribution actions.

近期提出的离线强化学习方法中，有很大一部分基于受约束或正则化的近似动态规划，例如 Q 学习或演员-评论器方法，其中约束或正则项用于限制策略偏离行为策略的程度。 我们将这些方法称为“多步动态规划”算法，因为它们经过多次迭代执行真正的动态规划，因此在获得高覆盖率数据时原则上能够恢复最优策略。 这些约束可以通过显式密度模型、隐式散度约束实现，也可以通过在策略改进目标中加入监督学习项来实现。 还有一些工作提出直接正则化 Q 函数，使其为分布外动作产生较低价值。

Our method is also a multi-step dynamic programming algorithm. However, in contrast to prior works, our method completely avoids directly querying the learned Q-function with unseen actions during training, removing the need for any constraint during this stage, though the subsequent policy extraction, which is based on advantage-weighted regression, does apply an implicit constraint. However, this policy does not actually influence value function training.

我们的方法同样是一种多步动态规划算法。 但与以往工作不同，我们的方法在训练期间完全避免使用未见动作直接查询学习到的 Q 函数，从而不需要在该阶段施加任何约束；不过，后续基于优势加权回归的策略提取确实会施加隐式约束。 然而，这个策略实际上并不影响价值函数训练。

In contrast to multi-step dynamic programming methods, several recent works have proposed methods that rely either on a single step of policy iteration, fitting the value function or Q-function of the behavior policy and then extracting the corresponding greedy policy, or else avoid value functions completely and utilize behavioral cloning-style objectives. We collectively refer to these as “single-step” approaches. These methods avoid needing to query unseen actions as well, since they either use no value function at all, or learn the value function of the behavior policy. Although these methods are simple to implement and effective on the MuJoCo locomotion tasks in D4RL, we show that such single-step methods perform very poorly on more complex datasets in D4RL, which require combining parts of suboptimal trajectories (“stitching”).

与多步动态规划方法不同，近期一些工作提出的方法要么仅依赖一步策略迭代，即拟合行为策略的价值函数或 Q 函数，再提取相应的贪心策略；要么完全避开价值函数，采用行为克隆风格的目标。 我们将它们统称为“单步”方法。 这些方法同样无需查询未见动作，因为它们或者完全不使用价值函数，或者学习行为策略的价值函数。 尽管这些方法易于实现，并且在 D4RL 的 MuJoCo 运动任务上有效，但我们表明，这类单步方法在更复杂的 D4RL 数据集上表现很差，因为这些数据集需要组合次优轨迹的不同片段，即进行“拼接”。

Prior multi-step dynamic programming methods perform much better in such settings, as does our method. We discuss this distinction in more detail in Section 5.1. Our method also shares the simplicity and computational efficiency of single-step approaches, providing an appealing combination of the strengths of both types of methods.

以往的多步动态规划方法在这类环境中表现好得多，我们的方法亦是如此。 我们在第 5.1 节中更详细地讨论这一区别。 我们的方法还具备单步方法的简洁性与计算效率，将两类方法的优势有吸引力地结合起来。

Our method is based on estimating the characteristics of a random variable. Several recent works involve approximating statistical quantities of the value function distribution. In particular, quantile regression has been previously used in reinforcement learning to estimate the quantile function of a state-action value function. Although our method is related, in that we perform expectile regression, our aim is not to estimate the distribution of values that results from stochastic transitions, but rather estimate expectiles of the state value function with respect to random actions.

我们的方法以估计随机变量的特征为基础。 近期有多项工作涉及逼近价值函数分布的统计量。 特别是，分位数回归此前已用于强化学习，以估计状态-动作价值函数的分位数函数。 虽然我们的方法与之相关，因为我们采用期望分位回归，但我们的目标并非估计随机转移所产生的价值分布，而是估计状态价值函数相对于随机动作的期望分位数。

This is a very different statistic: our aim is not to determine how the $Q$-value can vary with different future outcomes, but how the $Q$-value can vary with different actions while averaging together future outcomes due to stochastic dynamics. While prior work on distributional RL can also be used for offline RL, it would suffer from the same action extrapolation issues as other methods, and would require similar constraints or regularization, while our method does not.

这是一个非常不同的统计量：我们的目标不是确定 $Q$ 值如何随不同未来结果变化，而是确定 $Q$ 值如何随不同动作变化，同时对随机动态导致的未来结果取平均。 尽管分布式强化学习的既有工作也可用于离线强化学习，但它会遭遇与其他方法相同的动作外推问题，并需要类似的约束或正则化，而我们的方法不需要。

3. Preliminaries

The RL problem is formulated in the context of a Markov decision process (MDP) $(\mathcal{S},\mathcal{A},p_0(s),p(s^{\prime }|s,a),r(s,a),\gamma )$, where $\mathcal{S}$ is a state space, $\mathcal{A}$ is an action space, $p_0(s)$ is a distribution of initial states, $p(s^{\prime }|s,a)$ is the environment dynamics, $r(s,a)$ is a reward function, and $\gamma$ is a discount factor. The agent interacts with the MDP according to a policy $\pi (a|s)$. The goal is to obtain a policy that maximizes the cumulative discounted returns:

强化学习问题在马尔可夫决策过程（MDP）$(\mathcal{S},\mathcal{A},p_0(s),p(s^{\prime }|s,a),r(s,a),\gamma )$ 的背景下进行形式化，其中 $\mathcal{S}$ 是状态空间，$\mathcal{A}$ 是动作空间，$p_0(s)$ 是初始状态分布，$p(s^{\prime }|s,a)$ 是环境动态，$r(s,a)$ 是奖励函数，$\gamma$ 是折扣因子。 智能体按照策略 $\pi (a|s)$ 与 MDP 交互。 目标是获得一个使累积折扣回报最大化的策略：

$${\pi }^{\ast }=\arg \underset{\pi }{max}{\mathbb{E}}_{\begin{array}{c}{s}_0\sim p_0(\cdot ),a_t\sim \pi (\cdot |s_t)\\ s_{t+1}\sim p(\cdot |s_t,a_t)\end{array}}[\sum _{t=0}^{\mathrm{\infty }}{\gamma }^{t}r(s_t,a_t)].$$

Off-policy RL methods based on approximate dynamic programming typically utilize a state-action value function ($Q$-function), referred to as $Q(s,a)$, which corresponds to the discounted returns obtained by starting from the state $s$ and action $a$, and then following the policy $\pi$.

基于近似动态规划的离策略强化学习方法通常使用状态-动作价值函数（$Q$ 函数）$Q(s,a)$，它对应于从状态 $s$ 和动作 $a$ 开始、随后遵循策略 $\pi$ 所获得的折扣回报。

Offline reinforcement learning. In contrast to online (on-policy or off-policy) RL methods, offline RL uses previously collected data without any additional data collection. Like many recent offline RL methods, our work builds on approximate dynamic programming methods that minimize temporal difference error, according to the following loss:

离线强化学习。 与在线的同策略或离策略强化学习方法不同，离线强化学习使用先前收集的数据，而不进行任何额外的数据收集。 与近期许多离线强化学习方法一样，我们的工作建立在最小化时序差分误差的近似动态规划方法之上，其损失如下：

$$\begin{array}{}\text{(1)}& \begin{array}{rl}{L}_{\mathrm{TD}}(\theta )& ={\mathbb{E}}_{(s,a,s^{\prime })\sim \mathcal{D}}[\\ & (r(s,a)+\gamma \underset{a^{\prime }}{max}{Q}_{\hat{\theta }}(s^{\prime },a^{\prime })\\ & -Q_{\theta }(s,a){)}^2].\end{array}\end{array}$$

where $\mathcal{D}$ is the dataset, $Q_{\theta }(s,a)$ is a parameterized Q-function, $Q_{\hat{\theta }}(s,a)$ is a target network (e.g., with soft parameters updates defined via Polyak averaging), and the policy is defined as $\pi (s)={\text{argmax}}_a{Q}_{\theta }(s,a)$. Most recent offline RL methods modify either the value function loss (above) to regularize the value function in a way that keeps the resulting policy close to the data, or constrain the $\text{argmax}$ policy directly. This is important because out-of-distribution actions $a^{\prime }$ can produce erroneous values for $Q_{\hat{\theta }}(s^{\prime },a^{\prime })$ in the above objective, often leading to overestimation as the policy is defined to maximize the (estimated) Q-value.

其中，$\mathcal{D}$ 是数据集，$Q_{\theta }(s,a)$ 是参数化 Q 函数，$Q_{\hat{\theta }}(s,a)$ 是目标网络，例如通过 Polyak 平均定义软参数更新；策略定义为 $\pi (s)={\text{argmax}}_a{Q}_{\theta }(s,a)$。 近期大多数离线强化学习方法要么修改上述价值函数损失，以正则化价值函数并使所得策略接近数据，要么直接约束 $\text{argmax}$ 策略。 这一点很重要，因为分布外动作 $a^{\prime }$ 可能在上述目标中为 $Q_{\hat{\theta }}(s^{\prime },a^{\prime })$ 产生错误价值；由于策略被定义为最大化估计的 Q 值，这通常会导致高估。

4. Implicit Q-Learning

In this work, we aim to entirely avoid querying out-of-sample (unseen) actions in our TD loss. We start by considering fitted $Q$ evaluation with a SARSA-style objective, which simply aims to learn the value of the dataset policy ${\pi }_{\beta }$ (also called the behavior policy):

在本文中，我们旨在完全避免在时序差分损失中查询样本外的未见动作。 我们首先考虑采用 SARSA 风格目标的拟合 $Q$ 评估，其目的只是学习数据集策略 ${\pi }_{\beta }$（也称行为策略）的价值：

$$\begin{array}{}\text{(2)}& \begin{array}{rl}L(\theta )& ={\mathbb{E}}_{(s,a,s^{\prime },a^{\prime })\sim \mathcal{D}}[\\ & (r(s,a)+\gamma Q_{\hat{\theta }}(s^{\prime },a^{\prime })\\ & -Q_{\theta }(s,a){)}^2].\end{array}\end{array}$$

This objective never queries values for out-of-sample actions, in contrast to Equation (1). One specific property of this objective that is important for this work is that it uses mean squared error (MSE) that fits $Q_{\theta }(s,a)$ to predict the mean statistics of the TD targets. Thus, if we assume unlimited capacity and no sampling error, the optimal parameters should satisfy

与公式（1）不同，该目标从不查询样本外动作的价值。 这个目标有一个对本文很重要的特性：它使用均方误差（MSE）拟合 $Q_{\theta }(s,a)$，以预测时序差分目标的均值统计量。 因此，如果假设容量无限且不存在采样误差，最优参数应满足

$$\begin{array}{}\text{(3)}& Q_{{\theta }^{\ast }}(s,a)\approx r(s,a)+\gamma {\mathbb{E}}_{\begin{array}{c}{s}^{\prime }\sim p(\cdot |s,a)\\ a^{\prime }\sim {\pi }_{\beta }(\cdot |s^{\prime })\end{array}}[Q_{\hat{\theta }}(s^{\prime },a^{\prime })].\end{array}$$

Prior work has proposed directly using this objective to learn $Q^{{\pi }_{\beta }}$, and then train the policy ${\pi }_{\psi }$ to maximize $Q^{{\pi }_{\beta }}$. This avoids any issues with out-of-distribution actions, since the TD loss only uses dataset actions. However, while this procedure works well empirically on simple MuJoCo locomotion tasks in D4RL, we will show that it performs very poorly on more complex tasks that benefit from multi-step dynamic programming. In our method, which we derive next, we retain the benefits of using this SARSA-like objective, but modify it so that it allows us to perform multi-step dynamic programming and learn a near-optimal Q-function.

既有工作提出直接使用该目标学习 $Q^{{\pi }_{\beta }}$，再训练策略 ${\pi }_{\psi }$ 以最大化 $Q^{{\pi }_{\beta }}$。 由于时序差分损失只使用数据集中的动作，这避免了分布外动作带来的问题。 然而，尽管该过程在 D4RL 的简单 MuJoCo 运动任务上经验效果良好，我们将表明，它在受益于多步动态规划的更复杂任务上表现很差。 在下面推导的方法中，我们保留使用这种类 SARSA 目标的优势，同时对其进行修改，使其能够执行多步动态规划并学习近似最优的 Q 函数。

Our method will perform a $Q$-function update similar to Equation (2), but we will aim to estimate the maximum $Q$-value over actions that are in the support of the data distribution. Crucially, we will show that it is possible to do this without ever querying the learned Q-function on out-of-sample actions by utilizing expectile regression. Formally, the value function we aim to learn is given by:

我们的方法将执行类似公式（2）的 $Q$ 函数更新，但目标是估计数据分布支撑集内动作的最大 Q 值。 关键在于，我们将表明，利用期望分位回归可以做到这一点，且始终不在样本外动作上查询学习到的 Q 函数。 形式上，我们要学习的价值函数为：

$$\begin{array}{}\text{(4)}& L(\theta )={\mathbb{E}}_{(s,a,s^{\prime })\sim \mathcal{D}}\left[{(r(s,a)+\gamma \underset{\begin{array}{c}{a}^{\prime }\in \mathcal{A}\\ {\pi }_{\beta }(a^{\prime }|s^{\prime })>0\end{array}}{max}{Q}_{\hat{\theta }}(s^{\prime },a^{\prime })-Q_{\theta }(s,a))}^2\right].\end{array}$$

Our algorithm, implicit Q-Learning (IQL), aims to estimate this objective while evaluating the $Q$-function only on the state-action pairs in the dataset. To this end, we propose to fit $Q_{\theta }(s,a)$ to estimate state-conditional expectiles of the target values, and show that specific expectiles approximate the maximization defined above. In Section 4.4 we show that this approach performs multi-step dynamic programming in theory, and in Section 5.1 we show that it does so in practice.

我们的隐式 Q 学习（IQL）算法旨在估计这个目标，同时只在数据集中的状态-动作对上评估 $Q$ 函数。 为此，我们提出拟合 $Q_{\theta }(s,a)$ 以估计目标价值的状态条件期望分位数，并表明特定的期望分位数能够逼近上述最大化运算。 我们在第 4.4 节从理论上证明该方法执行多步动态规划，并在第 5.1 节中通过实践加以说明。

4.1 Expectile Regression

Practical methods for estimating various statistics of a random variable have been thoroughly studies in applied statistics and econometrics. The $\tau \in (0,1)$ expectile of some random variable $X$ is defined as a solution to the asymmetric least squares problem:

用于估计随机变量各种统计量的实用方法，已在应用统计学和计量经济学中得到深入研究。 随机变量 $X$ 的 $\tau \in (0,1)$ 期望分位数定义为以下非对称最小二乘问题的解：

$$\begin{array}{rl}{m}_{\tau }& =\arg \underset{m}{min}{\mathbb{E}}_{x\sim X}[L_2^{\tau }(x-m)],\\ L_2^{\tau }(u)& =|\tau -\mathbf{1}(u<0)|u^2.\end{array}$$

That is, for $\tau >0.5$, this asymmetric loss function downweights the contributions of $x$ values smaller than $m_{\tau }$ while giving more weights to larger values (see Figure 1, left). Expectile regression is closely related to quantile regression, which is a popular technique for estimating quantiles of a distribution widely used in reinforcement learning. The quantile regression loss is defined as an asymmetric ${\ell }_1$ loss.

也就是说，当 $\tau >0.5$ 时，这个非对称损失函数会降低小于 $m_{\tau }$ 的 $x$ 值的贡献权重，同时为较大值赋予更高权重（见图1左侧）。 期望分位回归与分位数回归密切相关，后者是一种估计分布分位数的常用技术，在强化学习中应用广泛。 分位数回归损失被定义为非对称 ${\ell }_1$ 损失。

图1：左：期望分位回归使用的非对称平方损失，τ = 0.5 对应标准均方误差损失，τ = 0.9 对正差异赋予更高权重。中：正态分布的期望分位数。右：估计二维随机变量状态条件期望分位数的示例；τ = 0.5 对应条件均值统计量，而 τ 接近 1 时逼近 y 的分布内最大值。

We can also use this formulation to predict expectiles of a conditional distribution:

我们也可以使用这一形式来预测条件分布的期望分位数：

$$\arg \underset{m_{\tau }(x)}{min}{\mathbb{E}}_{(x,y)\sim \mathcal{D}}[L_2^{\tau }(y-m_{\tau }(x))].$$

Figure 1 (right) illustrates conditional expectile regression on a simple two-dimensional distribution. Note that we can optimize this objective with stochastic gradient descent. It provides unbiased gradients and is easy to implement with standard machine learning libraries.

图1右侧展示了一个简单二维分布上的条件期望分位回归。 请注意，我们可以使用随机梯度下降优化该目标。 它提供无偏梯度，并且易于使用标准机器学习库实现。

4.2 Learning the Value Function with Expectile Regression

Expectile regression provides us with a powerful framework to estimate statistics of a random variable beyond mean regression. We can use expectile regression to modify the policy evaluation objective in Equation (2) to predict an upper expectile of the TD targets that approximates the maximum of $r(s,a)+\gamma Q_{\hat{\theta }}(s^{\prime },a^{\prime })$ over actions $a^{\prime }$ constrained to the dataset actions, as in Equation (4). This leads to the following expectile regression objective:

期望分位回归为我们提供了一个强大的框架，可以估计均值回归之外的随机变量统计量。 我们可以使用期望分位回归修改公式（2）中的策略评估目标，以预测时序差分目标的上侧期望分位数，从而逼近公式（4）中受限于数据集动作的 $a^{\prime }$ 上 $r(s,a)+\gamma Q_{\hat{\theta }}(s^{\prime },a^{\prime })$ 的最大值。 这得到以下期望分位回归目标：

$$\begin{array}{rl}L(\theta )={\mathbb{E}}_{(s,a,s^{\prime },a^{\prime })\sim \mathcal{D}}[L_2^{\tau }(& r(s,a)+\gamma Q_{\hat{\theta }}(s^{\prime },a^{\prime })\\ & -Q_{\theta }(s,a))].\end{array}$$

However, this formulation has a significant drawback. Instead of estimating expectiles just with respect to the actions in the support of the data, it also incorporates stochasticity that comes from the environment dynamics $s^{\prime }\sim p(\cdot |s,a)$. Therefore, a large target value might not necessarily reflect the existence of a single action that achieves that value, but rather a “lucky” sample that happened to have transitioned into a good state.

然而，这种形式存在一个显著缺点。 它不仅相对于数据支撑集中的动作估计期望分位数，还纳入了来自环境动态 $s^{\prime }\sim p(\cdot |s,a)$ 的随机性。 因此，一个较大的目标价值未必意味着存在一个能够达到该价值的动作，也可能只是一个碰巧转移到良好状态的“幸运”样本。

We resolve this by introducing a separate value function that approximates an expectile only with respect to the action distribution, leading to the following loss:

我们通过引入一个单独的价值函数来解决这一问题，该函数只相对于动作分布逼近期望分位数，从而得到以下损失：

$$\begin{array}{}\text{(5)}& L_V(\psi )={\mathbb{E}}_{(s,a)\sim \mathcal{D}}\left[L_2^{\tau }(Q_{\hat{\theta }}(s,a)-V_{\psi }(s))\right].\end{array}$$

We can then use this estimate to update the $Q$-functions with the MSE loss, which averages over the stochasticity from the transitions and avoids the “lucky” sample issue mentioned above:

随后，我们可以使用这一估计，通过均方误差损失更新 $Q$ 函数；该损失会对转移中的随机性取平均，从而避免上述“幸运”样本问题：

$$\begin{array}{}\text{(6)}& L_Q(\theta )={\mathbb{E}}_{(s,a,s^{\prime })\sim \mathcal{D}}\left[{(r(s,a)+\gamma V_{\psi }(s^{\prime })-Q_{\theta }(s,a))}^2\right].\end{array}$$

Note that these losses do not use any explicit policy, and only utilize actions from the dataset for both objectives, similarly to SARSA-style policy evaluation. In Section 4.4, we will show that this procedure recovers the optimal Q-function under some assumptions.

请注意，这些损失不使用任何显式策略，并且两个目标都只使用数据集中的动作，与 SARSA 风格的策略评估类似。 我们将在第 4.4 节中表明，在一些假设下，该过程能够恢复最优 Q 函数。

4.3 Policy Extraction and Algorithm Summary

Algorithm 1: Implicit Q-learning

1. Initialize parameters $\psi$, $\theta$, $\hat{\theta }$, $\varphi$.
2. TD learning (IQL):
3. for each gradient step do
4.   $\psi \leftarrow \psi -{\lambda }_V{\mathrm{\nabla }}_{\psi }{L}_V(\psi )$
5.   $\theta \leftarrow \theta -{\lambda }_Q{\mathrm{\nabla }}_{\theta }{L}_Q(\theta )$
6.   $\hat{\theta }\leftarrow (1-\alpha )\hat{\theta }+\alpha \theta$
7. end for
8. Policy extraction (AWR):
9. for each gradient step do
10.   $\varphi \leftarrow \varphi -{\lambda }_{\pi }{\mathrm{\nabla }}_{\varphi }{L}_{\pi }(\varphi )$
11. end for

While our modified TD learning procedure learns an approximation to the optimal Q-function, it does not explicitly represent the corresponding policy, and therefore requires a separate policy extraction step. In the spirit of preserving simplicity and efficiency, we aim for a simple method for policy extraction. As before, we aim to avoid using out-of-samples actions. Therefore, we extract the policy using advantage weighted regression:

虽然修改后的时序差分学习过程会学习最优 Q 函数的近似，但它没有显式表示相应策略，因此需要单独的策略提取步骤。 为了保持简洁与高效，我们希望采用一种简单的策略提取方法。 与前文一样，我们旨在避免使用样本外动作。 因此，我们使用优势加权回归提取策略：

$$\begin{array}{}\text{(7)}& L_{\pi }(\varphi )={\mathbb{E}}_{(s,a)\sim \mathcal{D}}[\exp (\beta (Q_{\hat{\theta }}(s,a)-V_{\psi }(s)))\log {\pi }_{\varphi }(a|s)].\end{array}$$

where $\beta \in [0,\mathrm{\infty })$ is an inverse temperature. For smaller hyperparameter values, the objective behaves similarly to behavioral cloning, while for larger values, it attempts to recover the maximum of the $Q$-function. As shown in prior work, this objective learns a policy that maximizes the $Q$-values subject to a distribution constraint.

其中，$\beta \in [0,\mathrm{\infty })$ 是逆温度参数。 当超参数取值较小时，该目标的行为类似于行为克隆；当取值较大时，它会尝试恢复 $Q$ 函数的最大值。 正如既有工作所示，该目标会学习一个在分布约束下最大化 Q 值的策略。

Our final algorithm consists of two stages. First, we fit the value function and $Q$, performing a number of gradient updates alternating between Eqn. (5) and (6). Second, we perform stochastic gradient descent on Equation (7). For both steps, we use a version of clipped double Q-learning, taking a minimum of two $Q$-functions for $V$-function and policy updates.

我们的最终算法包含两个阶段。 首先，我们拟合价值函数和 $Q$，在公式（5）与公式（6）之间交替执行若干次梯度更新。 其次，我们对公式（7）执行随机梯度下降。 在两个阶段中，我们都使用裁剪双 Q 学习的一个版本，在更新 $V$ 函数和策略时取两个 $Q$ 函数的较小值。

We summarize our final method in Algorithm 1. Note that the policy does not influence the value function in any way, and therefore extraction could be performed either concurrently or after TD learning. Concurrent learning provides a way to use IQL with online finetuning, as we discuss in Section 5.3.

我们在算法1中总结了最终方法。 请注意，策略不会以任何方式影响价值函数，因此既可以并行执行策略提取，也可以在时序差分学习之后执行。 正如我们在第 5.3 节中所讨论的，并行学习提供了一种将 IQL 用于在线微调的方法。

4.4 Analysis

In this section, we will show that IQL can recover the optimal value function under the dataset support constraints. First, we prove a simple lemma that we will then use to show how our approach can enable learning the optimal value function.

在本节中，我们将表明，IQL 能够在数据集支撑集约束下恢复最优价值函数。 首先，我们证明一个简单引理，随后用它说明我们的方法如何实现最优价值函数学习。

Lemma 1. Let $X$ be a real-valued random variable with a bounded support and supremum of the support is $x^{\ast }$. Then,

引理 1。 设 $X$ 是一个支撑集有界的实值随机变量，且该支撑集的上确界为 $x^{\ast }$。 则有

$$\underset{\tau \to 1}{lim}{m}_{\tau }=x^{\ast }.$$

One can show that expectiles of a random variable have the same supremum $x^{\ast }$. Moreover, for all ${\tau }_1$ and ${\tau }_2$ such that ${\tau }_1<{\tau }_2$, we get $m_{{\tau }_1}\le m_{{\tau }_2}$. Therefore, the limit follows from the properties of bounded monotonically non-decreasing functions.

可以证明，随机变量的期望分位数具有相同的上确界 $x^{\ast }$。 此外，对满足 ${\tau }_1<{\tau }_2$ 的任意 ${\tau }_1$ 和 ${\tau }_2$，都有 $m_{{\tau }_1}\le m_{{\tau }_2}$。 因此，由有界单调非递减函数的性质即可得到该极限。

In the following theorems, we show that under certain assumptions, our method indeed approximates the optimal state-action value $Q^{\ast }$ and performs multi-step dynamical programming. We first prove a technical lemma relating different expectiles of the Q-function, and then derive our main result regarding the optimality of our method.

在下面的定理中，我们表明，在特定假设下，我们的方法确实能够逼近最优状态-动作价值 $Q^{\ast }$ 并执行多步动态规划。 我们首先证明一个联系 Q 函数不同期望分位数的技术引理，随后推导有关该方法最优性的主要结果。

For the sake of simplicity, we introduce the following notation for our analysis. Let ${\mathbb{E}}_{x\sim X}^{\tau }[x]$ be a ${\tau }^{\text{th}}$ expectile of $X$ (e.g., ${\mathbb{E}}^{0.5}$ corresponds to the standard expectation). Then, we define $V_{\tau }(s)$ and $Q_{\tau }(s,a)$, which correspond to optimal solutions of Eqn. (5) and (6) correspondingly, recursively as:

为简化分析，我们引入以下记号。 令 ${\mathbb{E}}_{x\sim X}^{\tau }[x]$ 表示 $X$ 的第 $\tau$ 个期望分位数，例如 ${\mathbb{E}}^{0.5}$ 对应标准期望。 随后，我们递归定义分别对应公式（5）和公式（6）最优解的 $V_{\tau }(s)$ 与 $Q_{\tau }(s,a)$：

$$\begin{array}{rl}{V}_{\tau }(s)& ={\mathbb{E}}_{a\sim \mu (\cdot |s)}^{\tau }[Q_{\tau }(s,a)],\\ Q_{\tau }(s,a)& =r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}[V_{\tau }(s^{\prime })].\end{array}$$

Lemma 2. For all $s$, ${\tau }_1$ and ${\tau }_2$ such that ${\tau }_1<{\tau }_2$ we get

引理 2。 对任意 $s$、${\tau }_1$ 和 ${\tau }_2$，若 ${\tau }_1<{\tau }_2$，则有

$$V_{{\tau }_1}(s)\le V_{{\tau }_2}(s).$$

The proof follows the policy improvement proof. See the appendix.

该证明沿用策略改进证明。 详见附录。

Corollary. For any $\tau$ and $s$ we have

推论。 对任意 $\tau$ 和 $s$，有

$$V_{\tau }(s)\le \underset{\begin{array}{c}a\in \mathcal{A}\\ {\pi }_{\beta }(a|s)>0\end{array}}{max}{Q}^{\ast }(s,a),$$

where $V_{\tau }(s)$ is defined as above and $Q^{\ast }(s,a)$ is an optimal state-action value function constrained to the dataset and defined as

其中 $V_{\tau }(s)$ 如上定义，而 $Q^{\ast }(s,a)$ 是受数据集约束的最优状态-动作价值函数，其定义为

$$\begin{array}{r}{Q}^{\ast }(s,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(\cdot |s,a)}[\underset{\begin{array}{c}{a}^{\prime }\in \mathcal{A}\\ {\pi }_{\beta }(a^{\prime }|s^{\prime })>0\end{array}}{max}{Q}^{\ast }(s^{\prime },a^{\prime })].\end{array}$$

The proof follows from the observation that convex combination is smaller than maximum.

该证明来自凸组合小于最大值这一观察。

Theorem.

定理。

$$\underset{\tau \to 1}{lim}{V}_{\tau }(s)=\underset{\begin{array}{c}a\in \mathcal{A}\\ {\pi }_{\beta }(a|s)>0\end{array}}{max}{Q}^{\ast }(s,a).$$

Follows from combining Lemma 1 and the corollary. Therefore, for a larger value of $\tau <1$, we get a better approximation of the maximum. On the other hand, it also becomes a more challenging optimization problem. Thus, we treat $\tau$ as a hyperparameter.

该结论由引理 1 与上述推论结合得到。 因此，当 $\tau <1$ 取较大值时，我们能够更好地逼近最大值。 另一方面，优化问题也会变得更具挑战性。 因此，我们将 $\tau$ 视为超参数。

Due to the property discussed in the theorem we dub our method implicit Q-learning (IQL). We also emphasize that our value learning method defines the entire spectrum of methods between SARSA ($\tau =0.5$) and Q-Learning ($\tau \to 1$).

由于定理中讨论的性质，我们将该方法称为隐式 Q 学习（IQL）。 我们还要强调，我们的价值学习方法定义了介于 SARSA（$\tau =0.5$）与 Q 学习（$\tau \to 1$）之间的完整方法谱系。

5. Experimental Evaluation

Our experiments aim to evaluate our method comparatively, in contrast to prior offline RL methods, and in particular to understand how our approach compares both to single-step methods and multi-step dynamic programming approaches. We will first demonstrate the benefits of multi-step dynamic programming methods, such as ours, in contrast to single-step methods, showing that on some problems this difference can be extremely large. We will then compare IQL with state-of-the-art single-step and multi-step algorithms on the D4RL benchmark tasks, studying the degree to which we can learn effective policies using only the actions in the dataset.

我们的实验旨在将该方法与以往离线强化学习方法进行比较评估，特别是理解它与单步方法及多步动态规划方法的差异。 我们首先展示以我们的方法为代表的多步动态规划相较于单步方法的优势，并表明在一些问题上，这种差异可能极其巨大。 随后，我们在 D4RL 基准任务上将 IQL 与最先进的单步和多步算法进行比较，研究仅使用数据集中的动作能够学习到何种程度的有效策略。

We examine domains that contain near-optimal trajectories, where single-step methods perform well, as well as domains with no optimal trajectories at all, which require multi-step dynamic programming. Finally, we will study how IQL compares to prior methods when finetuning with online RL starting from an offline RL initialization.

我们既考察包含近似最优轨迹、因而单步方法表现良好的领域，也考察完全没有最优轨迹、必须使用多步动态规划的领域。 最后，我们研究从离线强化学习初始化出发、使用在线强化学习微调时，IQL 与以往方法的比较结果。

5.1 The Difference Between One-Step Policy Improvement and IQL

We first use a simple maze environment to illustrate the importance of multi-step dynamic programming for offline RL. The maze has a u-shape, a single start state, and a single goal state (see Figure 2). The agent receives a reward of 10 for entering the goal state and zero reward for all other transitions. With a probability of $0.25$, the agent transitions to a random state, and otherwise to the commanded state. The dataset consists of 1 optimal trajectory and 99 trajectories with uniform random actions. Due to a short horizon of the problem, we use $\gamma =0.9$.

我们首先使用一个简单的迷宫环境来说明多步动态规划对离线强化学习的重要性。 该迷宫呈 U 形，只有一个起始状态和一个目标状态（见图2）。 智能体进入目标状态时获得 10 的奖励，其他所有转移的奖励均为零。 智能体以 $0.25$ 的概率转移到随机状态，否则转移到指令指定的状态。 数据集包含 1 条最优轨迹和 99 条动作均匀随机的轨迹。 由于该问题的时域较短，我们使用 $\gamma =0.9$。

(a) toy maze MDP

(b) true optimal V*

(c) One-step Policy Eval.

(d) IQL

图2：在玩具 U 形迷宫环境上评估 IQL。（a）玩具迷宫 MDP；（b）真实最优价值函数；（c）单步策略评估；（d）IQL。当静态数据集被次优动作严重污染时，单步策略评估得到的价值函数会随着远离奖励状态而过快衰减。IQL 将 SARSA 风格评估与多步动态规划结合，学习到更接近最优值的价值函数并产生更好的策略。

Figure 2 (c, d) illustrates the difference between single-step methods, in this case represented by Onestep RL and IQL with $\tau =0.95$. Although states closer to the high reward state will still have higher values, these values decay much faster as we move further away than they would for the optimal value function, and the resulting policy is highly suboptimal. Since IQL (d) performs iterative dynamic programming, it correctly propagates the signal, and the values are no longer dominated by noise. The resulting value function closely matches the true optimal value function (b).

图2（c、d）展示了单步方法与 $\tau =0.95$ 的 IQL 之间的差异，其中单步方法以 Onestep RL 为代表。 虽然距离高奖励状态更近的状态仍然具有更高价值，但随着距离增加，这些价值的衰减速度远快于最优价值函数，所得策略也高度次优。 由于 IQL（d）执行迭代动态规划，它能够正确传播信号，价值也不再由噪声主导。 所得价值函数与真实最优价值函数（b）高度吻合。

5.2 Comparisons on Offline RL Benchmarks

表1：MuJoCo 运动任务与 Ant Maze 任务上的平均归一化分数。IQL 在需要动态规划的高难度 Ant Maze 任务上超过以往方法，并在运动任务上与最佳既有方法竞争。

Dataset	BC	10% BC	DT	AWAC	Onestep RL	TD3+BC	CQL	IQL (Ours)
halfcheetah-medium-v2	42.6	42.5	42.6	43.5	48.4	48.3	44.0	47.4
hopper-medium-v2	52.9	56.9	67.6	57.0	59.6	59.3	58.5	66.3
walker2d-medium-v2	75.3	75.0	74.0	72.4	81.8	83.7	72.5	78.3
halfcheetah-medium-replay-v2	36.6	40.6	36.6	40.5	38.1	44.6	45.5	44.2
hopper-medium-replay-v2	18.1	75.9	82.7	37.2	97.5	60.9	95.0	94.7
walker2d-medium-replay-v2	26.0	62.5	66.6	27.0	49.5	81.8	77.2	73.9
halfcheetah-medium-expert-v2	55.2	92.9	86.8	42.8	93.4	90.7	91.6	86.7
hopper-medium-expert-v2	52.5	110.9	107.6	55.8	103.3	98.0	105.4	91.5
walker2d-medium-expert-v2	107.5	109.0	108.1	74.5	113.0	110.1	108.8	109.6
locomotion-v2 total	466.7	666.2	672.6	450.7	684.6	677.4	698.5	692.4
antmaze-umaze-v0	54.6	62.8	59.2	56.7	64.3	78.6	74.0	87.5
antmaze-umaze-diverse-v0	45.6	50.2	53.0	49.3	60.7	71.4	84.0	62.2
antmaze-medium-play-v0	0.0	5.4	0.0	0.0	0.3	10.6	61.2	71.2
antmaze-medium-diverse-v0	0.0	9.8	0.0	0.7	0.0	3.0	53.7	70.0
antmaze-large-play-v0	0.0	0.0	0.0	0.0	0.0	0.2	15.8	39.6
antmaze-large-diverse-v0	0.0	6.0	0.0	1.0	0.0	0.0	14.9	47.5
antmaze-v0 total	100.2	134.2	112.2	107.7	125.3	163.8	303.6	378.0
total	566.9	800.4	784.8	558.4	809.9	841.2	1002.1	1070.4
kitchen-v0 total	154.5	–	–	–	–	–	144.6	159.8
adroit-v0 total	104.5	–	–	–	–	–	93.6	118.1
total+kitchen+adroit	825.9	–	–	–	–	–	1240.3	1348.3
runtime	10m	10m	960m	20m	≈ 20m*	20m	80m	20m

* 单步方法与多步方法难以直接比较；Onestep RL 原文使用了与其他方法显著不同的批大小和网络规模。表中报告原始超参数结果，以及使用可比超参数时的运行时间。

Next, we evaluate our approach on the D4RL benchmark in comparison to prior methods (see Table 1). The MuJoCo tasks in D4RL consist of the Gym locomotion tasks, the Ant Maze tasks, and the Adroit and Kitchen robotic manipulation environments. Some prior works, particularly those proposing one-step methods, focus entirely on the Gym locomotion tasks. However, these tasks include a significant fraction of near-optimal trajectories in the dataset.

接下来，我们在 D4RL 基准上将该方法与以往方法进行比较（见表1）。 D4RL 中的 MuJoCo 任务包括 Gym 运动任务、Ant Maze 任务以及 Adroit 和 Kitchen 机器人操作环境。 一些以往工作，特别是提出单步方法的工作，完全聚焦于 Gym 运动任务。 然而，这些任务的数据集中包含相当比例的近似最优轨迹。

In contrast, the Ant Maze tasks, especially the medium and large ones, contain very few or no near-optimal trajectories, making them very challenging for one-step methods. These domains require “stitching” parts of suboptimal trajectories that travel between different states to find a path from the start to the goal of the maze. As we will show, multi-step dynamic programming is essential in these domains. The Adroit and Kitchen tasks are comparatively less discriminating, and we found that most RL methods perform similarly to imitation learning in these domains. We therefore focus our analysis on the Gym locomotion and Ant Maze domains, but include full Adroit and Kitchen results in the appendix for completeness.

相比之下，Ant Maze 任务，特别是中型和大型任务，只包含很少甚至完全不包含近似最优轨迹，因此对单步方法非常具有挑战性。 这些领域需要“拼接”在不同状态之间行进的次优轨迹片段，以找到从起点到迷宫目标的路径。 正如我们将要展示的，多步动态规划在这些领域中至关重要。 Adroit 和 Kitchen 任务的区分能力相对较弱，我们发现大多数强化学习方法在这些领域中的表现与模仿学习相近。 因此，我们的分析聚焦于 Gym 运动和 Ant Maze 领域，同时在附录中给出完整的 Adroit 与 Kitchen 结果。

Comparisons and baselines. We compare to methods that are representative of both multi-step dynamic programming and one-step approaches. In the former category, we compare to CQL, TD3+BC, and AWAC. In the latter category, we compare to Onestep RL and Decision Transformers. We obtained the Decision Transformers results on Ant Maze subsets of D4RL tasks using the author-provided implementation and following authors instructions communicated over email. We obtained results for TD3+BC and Onestep RL (Exp. Weight) directly from the authors.

比较方法与基线。 我们与能够代表多步动态规划和单步方法的算法进行比较。 在前一类中，我们比较 CQL、TD3+BC 和 AWAC。 在后一类中，我们比较 Onestep RL 和 Decision Transformers。 我们使用作者提供的实现，并遵循作者通过电子邮件传达的说明，获得 Decision Transformers 在 D4RL Ant Maze 子集上的结果。 TD3+BC 和 Onestep RL（Exp. Weight）的结果则直接由相应作者提供。

Note that Chen et al. and Brandfonbrener et al. incorrectly report results for some prior methods, such as CQL, using the “-v0” environments. These generally produce lower scores than the “-v2” environments that these papers use for their own methods. We use the “-v2” environments for all methods to ensure a fair comparison, resulting in higher values for CQL. Because of this fix, our reported CQL scores are higher than all other prior methods. We obtained results for “-v2” datasets using an author-suggested implementation.

请注意，Chen 等人与 Brandfonbrener 等人错误地使用“-v0”环境报告了 CQL 等一些以往方法的结果。 这些环境通常比两篇论文为自身方法使用的“-v2”环境产生更低的分数。 为确保公平比较，我们对所有方法统一使用“-v2”环境，因此 CQL 的数值更高。 由于这一修正，我们报告的 CQL 分数高于其他既有工作中的报告值。 我们使用作者建议的实现获得“-v2”数据集上的结果。

On the Gym locomotion tasks (halfcheetah, hopper, walker2d), we find that IQL performs comparably to the best performing prior method, CQL. On the more challenging Ant Maze task, IQL outperforms CQL, and outperforms the one-step methods by a very large margin.

在 Gym 运动任务 halfcheetah、hopper 和 walker2d 上，我们发现 IQL 的表现与最佳既有方法 CQL 相当。 在更具挑战性的 Ant Maze 任务上，IQL 超过 CQL，并以很大优势超过单步方法。

Runtime. Our approach is also computationally faster than the baselines (see Table 1). For the baselines, we measure runtime for our reimplementations of the methods in JAX built on top of JAXRL, which are typically faster than the original implementations. For example, the original implementation of CQL takes more than 4 hours to perform 1M updates, while ours takes only 80 minutes.

运行时间。 我们的方法在计算上也快于基线（见表1）。 对于基线，我们测量基于 JAXRL、使用 JAX 重新实现的方法的运行时间，这些实现通常快于原始实现。 例如，CQL 的原始实现完成 100 万次更新需要超过 4 小时，而我们的重新实现只需 80 分钟。

Even so, IQL still requires about 4x less time than our reimplementation of CQL on average, and is comparable to the fastest prior one-step methods. We did not reimplement Decision Transformers due to their complexity and report runtime of the original implementation.

即便如此，IQL 平均仍比我们重新实现的 CQL 少用约 4 倍时间，并且与以往最快的单步方法相当。 由于 Decision Transformers 较为复杂，我们没有重新实现它，而是报告其原始实现的运行时间。

图3：对于需要动态规划，即进行‘拼接’的 Ant Maze 任务，估计较大的期望分位参数 τ 至关重要。

Effect of $\tau$ hyperparameter. We also demonstrate that it is crucial to compute a larger expectile on tasks that require “stitching” (see Figure 3). With larger values of $\tau$, our method approximates $Q$-learning better, leading to better performance on the Ant Maze tasks.

$\tau$ 超参数的影响。 我们还表明，在需要“拼接”的任务上，计算较大的期望分位数至关重要（见图3）。 随着 $\tau$ 增大，我们的方法能够更好地逼近 Q 学习，从而在 Ant Maze 任务上取得更好表现。

5.3 Online Fine-tuning after Offline RL

表2：在线微调结果，展示离线强化学习后的初始性能，以及进行 100 万步在线强化学习后的性能。所有任务中，IQL 微调后的性能都显著高于离线初始化；除 pen-binary-v0 外，其最终性能与 AWAC 或 CQL 中的最佳结果相当或更好。

Dataset	AWAC	CQL	IQL (Ours)
antmaze-umaze-v0	56.7 → 59.0	70.1 → 99.4	86.7 → 96.0
antmaze-umaze-diverse-v0	49.3 → 49.0	31.1 → 99.4	75.0 → 84.0
antmaze-medium-play-v0	0.0 → 0.0	23.0 → 0.0	72.0 → 95.0
antmaze-medium-diverse-v0	0.7 → 0.3	23.0 → 32.3	68.3 → 92.0
antmaze-large-play-v0	0.0 → 0.0	1.0 → 0.0	25.5 → 46.0
antmaze-large-diverse-v0	1.0 → 0.0	1.0 → 0.0	42.6 → 60.7
antmaze-v0 total	107.7 → 108.3	151.5 → 231.1	370.1 → 473.7
pen-binary-v0	44.6 → 70.3	31.2 → 9.9	37.4 → 60.7
door-binary-v0	1.3 → 30.1	0.2 → 0.0	0.7 → 32.3
relocate-binary-v0	0.8 → 2.7	0.1 → 0.0	0.0 → 31.0
hand-v0 total	46.7 → 103.1	31.5 → 9.9	38.1 → 124.0
total	154.4 → 211.4	182.8 → 241.0	408.2 → 597.7

The policies obtained by offline RL can often be improved with a small amount of online interaction. IQL is well-suited for online fine-tuning for two reasons. First, IQL has strong offline performance, as shown in the previous section, which provides a good initialization. Second, IQL implements a weighted behavioral cloning policy extraction step, which has previously been shown to allow for better online policy improvement compared to other types of offline constraints.

离线强化学习得到的策略通常可以通过少量在线交互得到改进。 IQL 非常适合在线微调，原因有二。 首先，如上一节所示，IQL 具有很强的离线性能，能够提供良好的初始化。 其次，IQL 实现了一个加权行为克隆策略提取步骤；既有研究表明，与其他类型的离线约束相比，这种方式能够实现更好的在线策略改进。

To evaluate the finetuning capability of various RL algorithms, we first run offline RL on each dataset, then run 1M steps of online RL, and then report the final performance. We compare to AWAC, which has been proposed specifically for online finetuning, and CQL, which showed the best performance among prior methods in our experiments in the previous section. Exact experimental details are provided in the appendix.

为了评估不同强化学习算法的微调能力，我们首先在每个数据集上运行离线强化学习，随后进行 100 万步在线强化学习，最后报告最终性能。 我们与专门为在线微调提出的 AWAC，以及在上一节实验中表现最佳的既有方法 CQL 进行比较。 准确的实验细节见附录。

We use the challenging Ant Maze D4RL domains, as well as the high-dimensional dexterous manipulation environments from Rajeswaran et al., which Nair et al. propose to use to study online adaptation with AWAC. Results are shown in Table 2. On the Ant Maze domains, IQL significantly outperforms both prior methods after online finetuning. CQL attains the second best score, while AWAC performs comparatively worse due to much weaker offline initialization.

我们使用高难度的 D4RL Ant Maze 领域，以及 Rajeswaran 等人提出的高维灵巧操作环境；Nair 等人建议使用后者来研究 AWAC 的在线适应。 结果见表2。 在 Ant Maze 领域，在线微调后的 IQL 显著超过两种既有方法。 CQL 取得第二高分，而 AWAC 由于离线初始化弱得多，表现相对较差。

On the dexterous hand tasks, IQL performs significantly better than AWAC on relocate-binary-v0, comparably on door-binary-v0, and slightly worse on pen-binary-v0, with the best overall score.

在灵巧手任务上，IQL 在 relocate-binary-v0 上显著优于 AWAC，在 door-binary-v0 上与其相当，在 pen-binary-v0 上略逊于它，但总体分数最高。

6. Conclusion

We presented implicit Q-Learning (IQL), a general algorithm for offline RL that completely avoids any queries to values of out-of-sample actions during training while still enabling multi-step dynamic programming. To our knowledge, this is the first method that combines both of these features. This has a number of important benefits. First, our algorithm is computationally efficient: we can perform 1M updates on one GTX1080 GPU in less than 20 minutes.

我们提出了隐式 Q 学习（IQL），这是一种通用的离线强化学习算法；它在训练期间完全避免查询样本外动作的价值，同时仍能执行多步动态规划。 据我们所知，这是首个同时结合这两项特性的方法。 这带来了多项重要优势。 首先，我们的算法计算高效：在一块 GTX1080 GPU 上，不到 20 分钟即可完成 100 万次更新。

Second, it is simple to implement, requiring only minor modifications over a standard SARSA-like TD algorithm, and performing policy extraction with a simple weighted behavioral cloning procedure resembling supervised learning. Finally, despite the simplicity and efficiency of this method, we show that it attains excellent performance across all of the tasks in the D4RL benchmark, matching the best prior methods on the MuJoCo locomotion tasks, and exceeding the state-of-the-art performance on the challenging ant maze environments, where multi-step dynamic programming is essential for good performance.

其次，它易于实现，只需对标准类 SARSA 时序差分算法做少量修改，并通过类似监督学习的简单加权行为克隆过程执行策略提取。 最后，尽管该方法简洁而高效，我们仍表明它在 D4RL 基准的所有任务上都取得了出色表现：在 MuJoCo 运动任务上追平最佳既有方法，并在必须依靠多步动态规划才能取得良好表现的高难度 Ant Maze 环境中超过此前最先进的性能。