---

An Image is Worth 16x16 Words:
Transformers for Image Recognition at Scale

Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.

https://github.com/google-research/vision_transformer

98260+

---

Abstract

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

尽管Transformer架构已成为自然语言处理任务的事实标准，但其在计算机视觉领域的应用仍然有限。在视觉领域，注意力机制要么与卷积网络结合使用，要么用于替换卷积网络的某些组件，同时保持其整体结构不变。我们证明，这种对CNN的依赖并非必要，直接将纯Transformer应用于图像块序列可以在图像分类任务上取得非常好的性能。当在大量数据上进行预训练，并迁移到多个中型或小型图像识别基准（ImageNet、CIFAR-100、VTAB等）上时，与最先进的卷积网络相比，视觉Transformer（ViT）能够获得优异的结果，同时所需的训练计算资源大幅减少。

Introduction

Self-attention-based architectures, in particular Transformers (Vaswani et al., 2017), have become the model of choice in natural language processing (NLP). The dominant approach is to pre-train on a large text corpus and then fine-tune on a smaller task-specific dataset (Devlin et al., 2019). Thanks to Transformers' computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters (Brown et al., 2020; Lepikhin et al., 2020). With the models and datasets growing, there is still no sign of saturating performance.

基于自注意力的架构，特别是Transformer，已成为自然语言处理中的首选模型。主流方法是在大型文本语料库上进行预训练，然后在较小的特定任务数据集上进行微调。得益于Transformer的计算效率和可扩展性，训练规模空前的模型（超过1000亿参数）已成为可能。随着模型和数据集不断增长，性能仍未出现饱和迹象。

In computer vision, however, convolutional architectures remain dominant (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016). Inspired by NLP successes, multiple works try combining CNN-like architectures with self-attention (Wang et al., 2018; Carion et al., 2020), some replacing the convolutions entirely (Ramachandran et al., 2019; Wang et al., 2020a). The latter models, while theoretically efficient, have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns. Therefore, in large-scale image recognition, classic ResNet-like architectures are still state of the art (Mahajan et al., 2018; Xie et al., 2020; Kolesnikov et al., 2020).

然而，在计算机视觉领域，卷积架构仍然占据主导地位。受NLP成功的启发，多项研究尝试将类似CNN的架构与自注意力相结合，有些甚至完全替换了卷积。后者虽然在理论上高效，但由于使用了特殊的注意力模式，尚未在现代硬件加速器上有效扩展。因此，在大规模图像识别任务中，经典的类ResNet架构仍然是最先进的。

Inspired by the Transformer scaling successes in NLP, we experiment with applying a standard Transformer directly to images, with the fewest possible modifications. To do so, we split an image into patches and provide the sequence of linear embeddings of these patches as an input to a Transformer. Image patches are treated the same way as tokens (words) in an NLP application. We train the model on image classification in supervised fashion.

受Transformer在NLP领域扩展成功的启发，我们尝试将标准Transformer直接应用于图像，并尽可能少做修改。为此，我们将图像分割成块，并将这些块的线性嵌入序列作为Transformer的输入。图像块的处理方式与NLP应用中的词元相同。我们以有监督的方式在图像分类任务上训练模型。

When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.

当在没有强正则化的中等规模数据集（如ImageNet）上进行训练时，这些模型的准确率适中，比同等规模的ResNet低几个百分点。这个看似令人沮丧的结果或许在意料之中：Transformer缺乏CNN固有的一些归纳偏置，如平移等变性和局部性，因此在训练数据不足时难以很好地泛化。

However, the picture changes if the models are trained on larger datasets (14M-300M images). We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained at sufficient scale and transferred to tasks with fewer datapoints. When pre-trained on the public ImageNet-21k dataset or in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks. In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks.

然而，当模型在更大规模的数据集（1400万到3亿张图像）上进行训练时，情况就发生了变化。我们发现大规模训练胜过了归纳偏置。我们的视觉Transformer在足够规模的预训练后，迁移到数据点较少的任务上取得了优异的结果。当在公开的ImageNet-21k数据集或内部的JFT-300M数据集上进行预训练后，ViT在多个图像识别基准上接近或超越了 SOTA。特别是，最佳模型在ImageNet上达到了88.55%的准确率，在ImageNet-ReaL上达到90.72%，在CIFAR-100上达到94.55%，在包含19个任务的VTAB套件上达到77.63%。

Related Work

Transformers were proposed by Vaswani et al. (2017) for machine translation, and have since become the state of the art method in many NLP tasks. Large Transformer-based models are often pre-trained on large corpora and then fine-tuned for the task at hand: BERT (Devlin et al., 2019) uses a denoising self-supervised pre-training task, while the GPT line of work uses language modeling as its pre-training task (Radford et al., 2018; 2019; Brown et al., 2020).

Transformer由Vaswani等人（2017）为机器翻译提出，此后成为许多NLP任务中最先进的方法。大型基于Transformer的模型通常在大规模语料库上进行预训练，然后针对当前任务进行微调：BERT（Devlin等人，2019）使用去噪自监督预训练任务，而GPT系列工作使用语言建模作为其预训练任务（Radford等人，2018；2019；Brown等人，2020）。

Naive application of self-attention to images would require that each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes. Thus, to apply Transformers in the context of image processing, several approximations have been tried in the past. Parmar et al. (2018) applied the self-attention only in local neighborhoods for each query pixel instead of globally. Such local multi-head dot-product self attention blocks can completely replace convolutions (Hu et al., 2019; Ramachandran et al., 2019; Zhao et al., 2020). In a different line of work, Sparse Transformers (Child et al., 2019) employ scalable approximations to global self-attention in order to be applicable to images. An alternative way to scale attention is to apply it in blocks of varying sizes (Weissenborn et al., 2019), in the extreme case only along individual axes (Ho et al., 2019; Wang et al., 2020a). Many of these specialized attention architectures demonstrate promising results on computer vision tasks, but require complex engineering to be implemented efficiently on hardware accelerators.

将自注意力直接应用于图像需要每个像素关注其他所有像素。由于像素数量的平方成本，这无法扩展到实际输入尺寸。因此，为了在图像处理中应用Transformer，过去尝试了几种近似方法。Parmar等人（2018）将自注意力仅应用于每个查询像素的局部邻域，而不是全局范围。这种局部多头点积自注意力块可以完全取代卷积（Hu等人，2019；Ramachandran等人，2019；Zhao等人，2020）。在另一条工作线中，Sparse Transformers（Child等人，2019）采用可扩展的近似方法来处理全局自注意力，以便适用于图像。另一种扩展注意力的方法是将其应用于不同大小的块中（Weissenborn等人，2019），极端情况下仅沿单个轴应用（Ho等人，2019；Wang等人，2020a）。许多这些专门化的注意力架构在计算机视觉任务上显示出有希望的结果，但需要复杂的工程才能在硬件加速器上高效实现。

Most related to ours is the model of Cordonnier et al. (2020), which extracts patches of size $2\times 2$ from the input image and applies full self-attention on top. This model is very similar to ViT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-art CNNs. Moreover, Cordonnier et al. (2020) use a small patch size of $2\times 2$ pixels, which makes the model applicable only to small-resolution images, while we handle medium-resolution images as well.

与我们的工作最相关的是Cordonnier等人（2020）的模型，该模型从输入图像中提取 $2\times 2$ 大小的块，并在其上应用完整的自注意力。该模型与ViT非常相似，但我们的工作更进一步，证明大规模预训练使得普通Transformer能够与最先进的CNN竞争（甚至更好）。此外，Cordonnier等人（2020）使用 $2\times 2$ 像素的小块大小，这使得该模型仅适用于小分辨率图像，而我们也能处理中等分辨率的图像。

There has also been a lot of interest in combining convolutional neural networks (CNNs) with forms of self-attention, e.g. by augmenting feature maps for image classification (Bello et al., 2019) or by further processing the output of a CNN using self-attention, e.g. for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or unified text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).

将卷积神经网络与各种形式的自注意力相结合也引起了广泛兴趣，例如通过为图像分类增强特征图（Bello等人，2019），或使用自注意力进一步处理CNN的输出，例如用于目标检测（Hu等人，2018；Carion等人，2020）、视频处理（Wang等人，2018；Sun等人，2019）、图像分类（Wu等人，2020）、无监督目标发现（Locatello等人，2020）或统一的文本-视觉任务（Chen等人，2020c；Lu等人，2019；Li等人，2019）。

Another recent related model is image GPT (iGPT) (Chen et al., 2020a), which applies Transformers to image pixels after reducing image resolution and color space. The model is trained in an unsupervised fashion as a generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy of 72% on ImageNet.

另一个近期相关的模型是图像GPT（iGPT）（Chen等人，2020a），它在降低图像分辨率和色彩空间后将Transformer应用于图像像素。该模型以无监督方式作为生成模型进行训练，然后可以对得到的表示进行微调或线性探测以用于分类，在ImageNet上达到了72%的最大准确率。

Our work adds to the increasing collection of papers that explore image recognition at larger scales than the standard ImageNet dataset. The use of additional data sources allows to achieve state-of-the-art results on standard benchmarks (Mahajan et al., 2018; Touvron et al., 2019; Xie et al., 2020). Moreover, Sun et al. (2017) study how CNN performance scales with dataset size, and Kolesnikov et al. (2020); Djolonga et al. (2020) perform an empirical exploration of CNN transfer learning from large scale datasets such as ImageNet-21k and JFT-300M. We focus on these two latter datasets as well, but train Transformers instead of ResNet-based models used in prior works.

我们的工作加入了越来越多的探索超出标准ImageNet数据集的更大规模图像识别的论文行列。使用额外的数据源可以在标准基准上达到 SOTA 结果（Mahajan等人，2018；Touvron等人，2019；Xie等人，2020）。此外，Sun等人（2017）研究了CNN性能如何随数据集大小扩展，而Kolesnikov等人（2020）；Djolonga等人（2020）对来自ImageNet-21k和JFT-300M等大规模数据集的CNN迁移学习进行了实证探索。我们也关注后两个数据集，但训练的是Transformer，而不是先前工作中使用的基于ResNet的模型。

Method

In model design we follow the original Transformer (Vaswani et al., 2017) as closely as possible. An advantage of this intentionally simple setup is that scalable NLP Transformer architectures – and their efficient implementations – can be used almost out of the box.

在模型设计上，我们尽可能遵循原始的Transformer（Vaswani等人，2017）。这种有意简化的设置的一个优点是，可扩展的NLP Transformer架构——及其高效实现——几乎可以开箱即用。

3.1 Vision Transformer (ViT)

An overview of the model is depicted in Figure 1. The standard Transformer receives as input a 1D sequence of token embeddings. To handle 2D images, we reshape the image $x\in {\mathbb{R}}^{H\times W\times C}$ into a sequence of flattened 2D patches $x_p\in {\mathbb{R}}^{N\times (P^2\cdot C)}$, where ((H, W)) is the resolution of the original image, $C$ is the number of channels, ((P, P)) is the resolution of each image patch, and $N=HW/P^2$ is the resulting number of patches, which also serves as the effective input sequence length for the Transformer. The Transformer uses constant latent vector size $D$ through all of its layers, so we flatten the patches and map to $D$ dimensions with a trainable linear projection (Eq. 1). We refer to the output of this projection as the patch embeddings.

3.1 视觉Transformer

模型概述如图1所示。标准Transformer接收一维的token嵌入序列作为输入。为了处理二维图像，我们将图像 $x\in {\mathbb{R}}^{H\times W\times C}$ 重塑为展平的二维块序列 $x_p\in {\mathbb{R}}^{N\times (P^2\cdot C)}$，其中 ((H, W)) 是原始图像的分辨率，$C$ 是通道数，((P, P)) 是每个图像块的分辨率，而 $N=HW/P^2$ 是得到的块数，同时也作为Transformer的有效输入序列长度。Transformer在其所有层中使用恒定的潜在向量大小 $D$，因此我们将这些块展平，并通过一个可训练的线性投影映射到 $D$ 维度（公式1）。我们将此投影的输出称为块嵌入。

Similar to BERT's [class] token, we prepend a learnable embedding to the sequence of embedded patches ($z_0^0=x_{\text{class}}$), whose state at the output of the Transformer encoder ($z_L^0$) serves as the image representation $y$ (Eq. 4). Both during pre-training and fine-tuning, a classification head is attached to $z_L^0$. The classification head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time.

类似于BERT的[class] token，我们在嵌入块序列前附加一个可学习的嵌入（$z_0^0=x_{\text{class}}$），其在Transformer编码器输出端的状态（$z_L^0$）作为图像表示 $y$（公式4）。在预训练和微调期间，一个分类头连接到 $z_L^0$。分类头在预训练时由一个带单隐藏层的MLP实现，在微调时由一个单线性层实现。

Position embeddings are added to the patch embeddings to retain positional information. We use standard learnable 1D position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.4). The resulting sequence of embedding vectors serves as input to the encoder.

位置嵌入被添加到块嵌入中以保留位置信息。我们使用标准的可学习一维位置嵌入，因为我们没有观察到使用更高级的二维感知位置嵌入能带来显著的性能提升（附录D.4）。得到的嵌入向量序列作为编码器的输入。

The Transformer encoder (Vaswani et al., 2017) consists of alternating layers of multiheaded self-attention (MSA, see Appendix A) and MLP blocks (Eq. 2, 3). Layernorm (LN) is applied before every block, and residual connections after every block (Wang et al., 2019; Baevski & Auli, 2019).

Transformer编码器（Vaswani等人，2017）由多头自注意力（MSA，见附录A）和MLP块的交替层组成（公式2、3）。层归一化（LN）应用于每个块之前，残差连接应用于每个块之后（Wang等人，2019；Baevski & Auli，2019）。

$$\begin{array}{rlrlr}{\mathbf{z}}_0& =[{\mathbf{x}}_{\text{class}};{\mathbf{x}}_p^1\mathbf{E};\cdots ;{\mathbf{x}}_p^N\mathbf{E}]+{\mathbf{E}}_{pos},& & \mathbf{E}\in {\mathbb{R}}^{(P^2\cdot C)\times D},{\mathbf{E}}_{pos}\in {\mathbb{R}}^{(N+1)\times D}& (1)\\ {\mathbf{z}}_{\ell }^{\prime }& =\text{MSA}(\text{LN}({\mathbf{z}}_{\ell -1}))+{\mathbf{z}}_{\ell -1},& & \ell =1\dots L& (2)\\ {\mathbf{z}}_{\ell }& =\text{MLP}(\text{LN}({\mathbf{z}}_{\ell }^{\prime }))+{\mathbf{z}}_{\ell }^{\prime },& & \ell =1\dots L& (3)\\ \mathbf{y}& =\text{LN}({\mathbf{z}}_L^0)& & & (4)\end{array}$$

The MLP contains two layers with a GELU non-linearity.

MLP包含两层，使用GELU非线性激活函数。

Inductive bias. We note that Vision Transformer has much less image-specific inductive bias than CNNs. In CNNs, locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer throughout the whole model. In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global. The two-dimensional neighborhood structure is used very sparingly: in the beginning of the model by cutting the image into patches and at fine-tuning time for adjusting the position embeddings for images of different resolution (as described below). Other than that, the position embeddings at initialization time carry no information about the 2D positions of the patches and all spatial relations between the patches have to be learned from scratch.

归纳偏置。 我们注意到，与CNN相比，视觉Transformer具有更少的图像特定归纳偏置。在CNN中，局部性、二维邻域结构和平移等变性被融入到整个模型的每一层。在ViT中，只有MLP层是局部且平移等变的，而自注意力层是全局的。二维邻域结构的使用非常有限：在模型开始时将图像切割成块，以及在微调时为不同分辨率的图像调整位置嵌入（如下所述）。除此之外，初始化时的位置嵌入不携带关于块二维位置的任何信息，块之间的所有空间关系都必须从头开始学习。

Hybrid Architecture. As an alternative to raw image patches, the input sequence can be formed from feature maps of a CNN (LeCun et al., 1989). In this hybrid model, the patch embedding projection $E$ (Eq. 1) is applied to patches extracted from a CNN feature map. As a special case, the patches can have spatial size 1x1, which means that the input sequence is obtained by simply flattening the spatial dimensions of the feature map and projecting to the Transformer dimension. The classification input embedding and position embeddings are added as described above.

混合架构。 作为原始图像块的替代方案，输入序列可以由CNN的特征图形成（LeCun等人，1989）。在这种混合模型中，块嵌入投影 $E$（公式1）应用于从CNN特征图中提取的块。作为一种特殊情况，块的空间大小可以是1x1，这意味着输入序列是通过简单地将特征图的空间维度展平并投影到Transformer维度而获得的。分类输入嵌入和位置嵌入的添加方式与前述相同。

3.2 Fine-tuning and Higher Resolution

Typically, we pre-train ViT on large datasets, and fine-tune to (smaller) downstream tasks. For this, we remove the pre-trained prediction head and attach a zero-initialized $D\times K$ feedforward layer, where $K$ is the number of downstream classes. It is often beneficial to fine-tune at higher resolution than pre-training (Touvron et al., 2019; Kolesnikov et al., 2020). When feeding images of higher resolution, we keep the patch size the same, which results in a larger effective sequence length. The Vision Transformer can handle arbitrary sequence lengths (up to memory constraints), however, the pre-trained position embeddings may no longer be meaningful. We therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image. Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer.

3.2 微调与更高分辨率

通常，我们在大型数据集上预训练ViT，然后针对（较小的）下游任务进行微调。为此，我们移除预训练的预测头，并附加一个零初始化的 $D\times K$ 前馈层，其中 $K$ 是下游类别的数量。在比预训练时更高的分辨率下进行微调通常是有益的（Touvron等人，2019；Kolesnikov等人，2020）。当输入更高分辨率的图像时，我们保持块大小不变，这将导致更大的有效序列长度。视觉Transformer可以处理任意序列长度（受限于内存），然而，预训练的位置嵌入可能不再有意义。因此，我们根据预训练位置嵌入在原始图像中的位置，对其进行二维插值。请注意，这种分辨率调整和块提取是唯二将关于图像二维结构的归纳偏置手动注入到视觉Transformer中的点。

Conclusion

We have explored the direct application of Transformers to image recognition. Unlike prior works using self-attention in computer vision, we do not introduce image-specific inductive biases into the architecture apart from the initial patch extraction step. Instead, we interpret an image as a sequence of patches and process it by a standard Transformer encoder as used in NLP. This simple, yet scalable, strategy works surprisingly well when coupled with pre-training on large datasets. Thus, Vision Transformer matches or exceeds the state of the art on many image classification datasets, whilst being relatively cheap to pre-train.

我们探索了将Transformer直接应用于图像识别。与先前在计算机视觉中使用自注意力的工作不同，除了初始的图像块提取步骤外，我们没有在架构中引入图像特定的归纳偏置。相反，我们将图像解释为一系列图像块，并使用NLP中使用的标准Transformer编码器对其进行处理。这种简单但可扩展的策略，在与大规模数据集上的预训练相结合时，效果出奇地好。因此，视觉Transformer在许多图像分类数据集上达到或超越了 SOTA，同时预训练成本相对较低。

While these initial results are encouraging, many challenges remain. One is to apply ViT to other computer vision tasks, such as detection and segmentation. Our results, coupled with those in Carion et al. (2020), indicate the promise of this approach. Another challenge is to continue exploring self-supervised pre-training methods. Our initial experiments show improvement from self-supervised pre-training, but there is still large gap between self-supervised and large-scale supervised pre-training. Finally, further scaling of ViT would likely lead to improved performance.

虽然这些初步结果令人鼓舞，但许多挑战依然存在。其一是将ViT应用于其他计算机视觉任务，例如检测和分割。我们的结果，连同Carion等人（2020）的结果，表明了这种方法的潜力。另一个挑战是继续探索自监督预训练方法。我们的初步实验显示自监督预训练带来了改进，但自监督与大规模监督预训练之间仍然存在巨大差距。最后，进一步扩展ViT很可能会带来性能的提升。