---

SAM 3: Segment Anything with Concepts

Segmentation11000+620+ICLR 2026CCF-AMeta

Carion N, Gustafson L, Hu Y T, et al. SAM 3: Segment Anything with Concepts. ICLR 2026.

https://openreview.net/forum?id=r35clVtGzw

https://github.com/facebookresearch/sam3

---

SAM 3：用概念分割万物

Abstract

We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., “yellow school bus”), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.

我们提出 Segment Anything Model（SAM）3，这是一个统一模型，可根据概念提示检测、分割和跟踪图像与视频中的目标；概念提示可以是短名词短语（例如“黄色校车”）、图像示例，也可以是二者的组合。 可提示概念分割（Promptable Concept Segmentation，PCS）接收此类提示，并返回所有匹配目标实例的分割掩码和唯一标识。 为推进 PCS，我们构建了一个可扩展的数据引擎，在图像和视频上生成包含 400 万个独特概念标签（包括困难负样本）的高质量数据集。 我们的模型由共享同一骨干网络的图像级检测器和基于记忆的视频跟踪器组成。 我们通过存在性头将识别与定位解耦，从而提高检测准确率。 SAM 3 在图像和视频 PCS 上均将现有系统的准确率提高到两倍，并改进了此前 SAM 在视觉分割任务上的能力。 我们开源 SAM 3，并同时发布用于可提示概念分割的新基准 Segment Anything with Concepts（SA-Co）。

1. Introduction

图1：SAM 3 改进了 SAM 2 基于点击的可提示视觉分割能力（左），并引入新的可提示概念分割能力（右）。用户可以通过短名词短语、正负图像示例或二者组合，分割某一视觉概念的所有实例。

The ability to find and segment anything in a visual scene is foundational for multimodal AI, powering applications in robotics, content creation, augmented reality, data annotation, and broader sciences. The SAM series introduced the promptable segmentation task for images and videos, focusing on Promptable Visual Segmentation (PVS) with points, boxes or masks to segment a single object per prompt. While these methods achieved a breakthrough, they did not address the general task of finding and segmenting all instances of a concept appearing anywhere in the input (e.g., all “cats” in a video).

在视觉场景中发现并分割万物的能力是多模态人工智能的基础，它支撑着机器人、内容创作、增强现实、数据标注和更广泛科学领域中的应用。 SAM 系列提出了面向图像和视频的可提示分割任务，重点研究可提示视觉分割（Promptable Visual Segmentation，PVS），即通过点、框或掩码提示，每个提示分割一个目标。 尽管这些方法取得了突破，但它们没有解决一项更通用的任务：发现并分割输入中任意位置出现的某一概念的全部实例（例如视频中的所有“猫”）。

To fill this gap, we present SAM 3, a model that achieves a step change in promptable segmentation in images and videos, improving PVS relative to SAM 2 and setting a new standard for Promptable Concept Segmentation (PCS). We formalize the PCS task (Section 2) as taking text and/or image exemplars as input, and predicting instance and semantic masks for every single object matching the concept, while preserving object identities across video frames (see Figure 1). To focus on recognizing atomic visual concepts, we constrain text to simple noun phrases (NPs) such as “red apple” or “striped cat”. While SAM 3 is not designed for long referring expressions or queries requiring reasoning, we show that it can be straightforwardly combined with a Multimodal Large Language Model (MLLM) to handle more complex language prompts. Consistent with previous SAM versions, SAM 3 is fully interactive, allowing users to resolve ambiguities by adding refinement prompts to guide the model towards their intended output.

为填补这一空白，我们提出 SAM 3；该模型使图像和视频中的可提示分割能力实现跃升，不仅相较 SAM 2 改进了 PVS，还为*可提示概念分割（PCS）*树立了新标准。 我们将 PCS 任务（第 2 节）形式化为：接收文本和/或图像示例作为输入，为与概念匹配的每一个目标预测实例掩码和语义掩码，同时在视频帧之间保留目标身份（见图1）。 为专注于识别原子视觉概念，我们将文本限制为“红苹果”或“条纹猫”等简单名词短语（NP）。 尽管 SAM 3 并非为长指代表达式或需要推理的查询而设计，但我们表明，它可以直接与多模态大语言模型（MLLM）结合，以处理更复杂的语言提示。 与此前的 SAM 版本一致，SAM 3 支持完整交互，用户可以添加细化提示来消除歧义，引导模型得到预期输出。

图2：SAM 3 在 SA-Co 基准上相较 OWLv2 改进开放词汇概念分割的示例。

Our model (Section 3) consists of a detector and a tracker that share a vision encoder. The detector is a DETR-based model conditioned on text, geometry, and image exemplars. To address the challenge of open-vocabulary concept detection, we introduce a separate presence head to decouple recognition and localization, which is especially effective when training with challenging negative phrases. The tracker inherits the SAM 2 transformer encoder-decoder architecture, supporting video segmentation and interactive refinement. The decoupled design for detection and tracking avoids task conflict, as the detector needs to be identity agnostic, while the tracker's main objective is to separate identities in the video.

我们的模型（第 3 节）由共享视觉编码器的检测器和跟踪器组成。 检测器基于 DETR，并以文本、几何信息和图像示例为条件。 为应对开放词汇概念检测的挑战，我们引入独立的存在性头来解耦识别和定位；在使用具有挑战性的负短语训练时，它尤其有效。 跟踪器沿用 SAM 2 的 Transformer 编码器—解码器架构，支持视频分割和交互式细化。 检测与跟踪的解耦设计避免了任务冲突：检测器需要与身份无关，而跟踪器的主要目标是在视频中区分不同身份。

To unlock major performance gains, we build a human- and model-in-the-loop data engine (Section 4) that annotates a large and diverse training dataset. We innovate upon prior data engines in three key ways: (i) media curation: we curate more diverse media domains than past approaches that rely on homogeneous web sources, (ii) label curation: we significantly increase label diversity and difficulty by leveraging an ontology and multimodal LLMs as “AI annotators” to generate noun phrases and hard negatives, (iii) label verification: we double annotation throughput by fine-tuning MLLMs to be effective “AI verifiers” that achieve near-human accuracy. Starting from noisy media-phrase-mask pseudo-labels, our data engine checks mask quality and exhaustivity using both human and AI verifiers, filtering out correctly labeled examples and identifying challenging error cases. Human annotators then focus on fixing these errors by manually correcting masks. This enables us to annotate high-quality training data with 4M unique phrases and 52M masks, and a synthetic dataset with 38M phrases and 1.4B masks. We additionally create the Segment Anything with Concepts (SA-Co) benchmark for PCS (Section 5) containing 207K unique concepts with exhaustive masks in 120K images and 1.7K videos, $>50\times$ more concepts than existing benchmarks.

为释放显著的性能增益，我们构建了一个人类和模型共同参与的数据引擎（第 4 节），用于标注大规模且多样化的训练数据集。 相较以往的数据引擎，我们在三个关键方面作出创新：（i）媒体策展：与依赖同质化网络来源的既有方法相比，我们策展了更多样的媒体领域；（ii）标签策展：利用本体和作为“人工智能标注员”的多模态 LLM 生成名词短语及困难负样本，显著提升标签的多样性和难度；（iii）标签验证：通过微调 MLLM，使其成为准确率接近人类的有效“人工智能验证器”，将标注吞吐量提高一倍。 数据引擎从带噪声的媒体—短语—掩码伪标签开始，使用人类验证器和人工智能验证器检查掩码质量及穷尽性，过滤标注正确的样本，并识别具有挑战性的错误案例。 随后，人类标注员集中精力通过手动修正掩码来修复这些错误。 这使我们能够标注包含 400 万个独特短语和 5200 万个掩码的高质量训练数据，以及包含 3800 万个短语和 14 亿个掩码的合成数据集。 此外，我们还创建用于 PCS 的 Segment Anything with Concepts（SA-Co）基准（第 5 节），其中 12 万张图像和 1700 个视频包含 20.7 万个独特概念及穷尽掩码，概念数量是现有基准的 $>50\times$。

Our experiments (Section 6) show that SAM 3 sets a new state-of-the-art in promptable segmentation, e.g., reaching a zero-shot mask AP of 48.8 on LVIS vs. the current best of 38.5, surpassing baselines on our new SA-Co benchmark by at least $2\times$ (see examples in Figure 2), and improving upon SAM 2 on visual prompts. Ablations verify that the choice of backbone, novel presence head, and adding hard negatives all boost results, and establish scaling laws on the PCS task for both our high-quality and synthetic datasets. We open-source the SA-Co benchmark and release the SAM 3 checkpoints and inference code. On an H200 GPU, SAM 3 runs in 30 ms for a single image with 100+ detected objects. In video, the inference latency scales with the number of objects, sustaining near real-time performance for $\sim$ 5 concurrent objects. We review related work in Section 7; next, we dive into the task.

我们的实验（第 6 节）表明，SAM 3 在可提示分割方面建立了新的最先进水平，例如在 LVIS 上达到 48.8 的零样本掩码 AP，而当前最佳结果为 38.5；它在新 SA-Co 基准上以至少 $2\times$ 的幅度超过基线（示例见图2），并改进了 SAM 2 对视觉提示的处理能力。 消融实验验证了骨干网络选择、新型存在性头以及困难负样本均能改善结果，并分别针对高质量数据和合成数据建立了 PCS 任务上的缩放规律。 我们开源 SA-Co 基准，并发布 SAM 3 检查点和推理代码。 在 H200 GPU 上，SAM 3 处理一张包含 100 多个检测目标的图像仅需 30 ms。 在视频中，推理延迟随目标数量增长；对于同时存在的约 5 个目标，仍可保持接近实时的性能。 我们将在第 7 节回顾相关工作；下面深入介绍该任务。

2. Promptable Concept Segmentation (PCS)

图3：PCS 任务支持的初始提示与可选交互式细化提示。

We define the Promptable Concept Segmentation task as follows: given an image or short video ($\le$ 30 secs), detect, segment and track all instances of a visual concept specified by a short text phrase, image exemplars, or a combination of both. We restrict concepts to those defined by simple noun phrases (NPs) consisting of a noun and optional modifiers. Noun-phrase prompts (when provided) are global to all frames of the image/video, while image exemplars can be provided on individual frames as positive or negative bounding boxes to iteratively refine the target masks (see Figure 3).

我们将可提示概念分割任务定义如下：给定一张图像或一个短视频（$\le$ 30 秒），检测、分割并跟踪由短文本短语、图像示例或二者组合指定的视觉概念的所有实例。 我们将概念限制为由名词和可选修饰语组成的简单名词短语（NP）。 名词短语提示（若提供）对图像/视频的所有帧均为全局提示，而图像示例可以作为正边界框或负边界框提供在单独帧上，以迭代方式细化目标掩码（见图3）。

All prompts must be consistent in their category definition, or the model's behavior is undefined; e.g., “fish” cannot be refined with subsequent exemplar prompts of just the tail; instead the text prompt should be updated. Exemplar prompts are particularly useful when the model initially misses some instances, or when the concept is rare. Our vocabulary includes any simple noun phrase groundable in a visual scene, which makes the task intrinsically ambiguous. There can be multiple interpretations of phrases arising from polysemy (“mouse” device vs. animal), subjective descriptors (“cozy”, “large”), vague or context-dependent phrases that may not even be groundable (“brand identity”), boundary ambiguity (whether ‘mirror’ includes the frame) and factors such as occlusion and blur that obscure the extent of the object. While similar issues appear in large closed-vocabulary corpora (e.g., LVIS), they are alleviated by carefully curating the vocabulary and setting a clear definition of all the classes of interest. We address the ambiguity problem by collecting test annotations from three experts, adapting the evaluation protocol to allow multiple valid interpretations, designing the data pipeline/guidelines to minimize ambiguity in annotation, and an ambiguity module in the model.

所有提示的类别定义必须一致，否则模型行为没有明确定义；例如，不能先用“鱼”作为提示，再用仅包含鱼尾的示例提示进行细化，而应更新文本提示。 当模型最初漏掉某些实例或概念较为罕见时，示例提示尤其有用。 我们的词汇表包含可在视觉场景中定位的任意简单名词短语，这使任务本身具有歧义。 短语可能存在多种解释，包括一词多义（“mouse”指设备或动物）、主观描述词（“舒适”“大型”）、甚至无法定位的模糊或依赖上下文的短语（“品牌形象”）、边界歧义（“镜子”是否包含镜框），以及遮挡和模糊等妨碍确定目标范围的因素。 尽管大型封闭词汇语料库（例如 LVIS）中也会出现类似问题，但可以通过仔细策展词汇表并清晰定义所有关注类别来缓解。 我们通过收集三位专家的测试标注、调整评估协议以允许多种有效解释、设计数据流水线和指南以尽量减少标注歧义，以及在模型中加入歧义模块来处理这一问题。

3. Model

图4：SAM 3 架构概览。检测器和跟踪器共享感知编码器骨干网络，并通过检测、传播、匹配和记忆库共同处理视频中的概念分割。

SAM 3 is a generalization of SAM 2, supporting the new PCS task (Section 2) as well as the PVS task. It takes concept prompts (simple noun phrases, image exemplars) or visual prompts (points, boxes, masks) to define the objects to be (individually) segmented spatio-temporally. Image exemplars and visual prompts can be iteratively added on individual frames to refine the target masks---false positive and false negative objects can be removed or added respectively using image exemplars and an individual mask(let) can be refined using PVS in the style of SAM 2. Our architecture is broadly based on the SAM and (M)DETR series. Figure 4 shows the SAM 3 architecture, consisting of a dual encoder-decoder transformer---a detector for image-level capabilities---which is used in combination with a tracker and memory for video. The detector and tracker ingest vision-language inputs from an aligned Perception Encoder (PE) backbone. We present an overview below, see the appendix for details.

SAM 3 是 SAM 2 的推广，既支持新的 PCS 任务（第 2 节），也支持 PVS 任务。 它接收概念提示（简单名词短语、图像示例）或视觉提示（点、框、掩码），以定义需要在时空上逐个分割的目标。 可以在单独帧上迭代添加图像示例和视觉提示以细化目标掩码：分别使用图像示例移除假阳性目标或添加假阴性目标，并且可以按照 SAM 2 的方式使用 PVS 细化某个单独掩码（序列）。 我们的架构总体上基于 SAM 和（M）DETR 系列。 图4展示了 SAM 3 架构：它包含一个负责图像级能力的双编码器—解码器 Transformer 检测器，并与用于视频的跟踪器和记忆相结合。 检测器和跟踪器都从对齐的感知编码器（Perception Encoder，PE）骨干网络接收视觉—语言输入。 下面给出概览，详细信息见附录。

Detector Architecture. The architecture of the detector follows the general DETR paradigm. The image and text prompt are first encoded by PE and image exemplars, if present, are encoded by an exemplar encoder. We refer to the image exemplar tokens and text tokens jointly as “prompt tokens”. The fusion encoder then accepts the unconditioned embeddings from the image encoder and conditions them by cross-attending to the prompt tokens. The fusion is followed by a DETR-like decoder, where learned object queries cross-attend to the conditioned image embeddings from the fusion encoder. Each decoder layer predicts a classification logit for each object query (in our case, a binary label of whether the object corresponds to the prompt), and a delta from the bounding box predicted by the previous level. We use box-region-positional bias to help focalize the attention on each object, but unlike recent DETR models, we stick to vanilla attention. During training, we adopt dual supervision from DAC-DETR, and the Align loss. The mask head is adapted from MaskFormer. In addition, we also have a semantic segmentation head, which predicts a binary label for every pixel in the image, indicating whether or not it corresponds to the prompt. See the appendix for details.

检测器架构。 检测器架构遵循通用 DETR 范式。 首先由 PE 编码图像和文本提示；如果存在图像示例，则由示例编码器对其编码。 我们将图像示例 token 和文本 token 统称为“提示 token”。 随后，融合编码器接收图像编码器产生的无条件嵌入，并通过对提示 token 的交叉注意力为这些嵌入添加条件。 融合之后是一个类似 DETR 的解码器，其中学习得到的目标查询对融合编码器输出的条件图像嵌入执行交叉注意力。 每个解码器层为每个目标查询预测一个分类 logit（在本文中，是目标是否与提示对应的二元标签），以及相对上一层预测边界框的增量。 我们使用边界框区域位置偏置帮助注意力聚焦于各个目标，但与近期 DETR 模型不同，我们仍采用普通注意力。 训练期间，我们采用 DAC-DETR 的双重监督和 Align 损失。 掩码头由 MaskFormer 改造而来。 此外，我们还设置了语义分割头，为图像中的每个像素预测一个二元标签，指示其是否与提示对应。 详细信息见附录。

Presence Token. It can be difficult for each of the proposal queries to both recognize (what) and localize (where) an object in the image/frame. For the recognition component, contextual cues from the entire image are important. However, forcing proposal queries to understand the global context can be counterproductive, as it conflicts with the inherently local nature of the localization objective. We decouple the recognition and localization steps by introducing a learned global presence token. This token is solely responsible for predicting whether the target concept in the form of a noun phrase (NP) is present in the image/frame, i.e. $p(\text{NP is present in input})$. Each proposal query $q_i$ only needs to solve the localization problem $p(q_i\text{is a match}\mid \text{NP is present in input})$. The final score for each proposal query is the product of its own score and the presence score.

存在性 token。 对每个候选查询而言，同时识别图像/帧中的目标是什么以及定位它在哪里可能十分困难。 对于识别部分，来自整张图像的上下文线索非常重要。 然而，强制候选查询理解全局上下文可能适得其反，因为这与定位目标固有的局部性质相冲突。 我们引入学习得到的全局存在性 token，将识别步骤和定位步骤解耦。 该 token 仅负责预测以名词短语（NP）形式给出的目标概念是否存在于图像/帧中，即 $p(\text{输入中存在 NP})$。 每个候选查询 $q_i$ 只需解决定位问题 $p(q_i\text{匹配}\mid \text{输入中存在 NP})$。 每个候选查询的最终分数是其自身分数与存在性分数的乘积。

Image Exemplars and Interactivity. SAM 3 supports image exemplars, given as a pair---a bounding box and an associated binary label (positive or negative)---which can be used in isolation or to supplement the text prompt. The model then detects all the instances that match the prompt. For example, given a positive bounding box on a dog, the model will detect all dogs in the image. This is different from the PVS task in SAM 1 and 2, where a visual prompt yields only a single object instance. Each image exemplar is encoded separately by the exemplar encoder using an embedding for the position, an embedding for the label, and ROI-pooled visual features, then concatenated and processed by a small transformer. The resulting prompt is concatenated to the text prompt to comprise the prompt tokens. Image exemplars can be interactively provided based on errors in current detections to refine the output.

图像示例与交互性。 SAM 3 支持由边界框和相应二元标签（正或负）组成的图像示例；它既可单独使用，也可作为文本提示的补充。 随后，模型检测与提示匹配的所有实例。 例如，给定狗的正边界框，模型将检测图像中的所有狗。 这不同于 SAM 1 和 SAM 2 中的 PVS 任务；在 PVS 中，一个视觉提示只产生一个目标实例。 示例编码器分别使用位置嵌入、标签嵌入和经过 ROI 池化的视觉特征编码每个图像示例，再将它们拼接并由一个小型 Transformer 处理。 得到的提示与文本提示拼接，共同构成提示 token。 可以根据当前检测中的错误交互式提供图像示例，以细化输出。

Tracker and Video Architecture. Given a video and a prompt $P$, we use the detector and a tracker (see Figure 4) to detect and track objects corresponding to the prompt throughout the video. On each frame, the detector finds new objects ${\mathcal{O}}_t$ and the tracker propagates masklets ${\mathcal{M}}_{t-1}$ (spatial-temporal masks) from frames at the previous time $t-1$ to their new locations ${\hat{\mathcal{M}}}_t$ on the current frame at time $t$. We use a matching function to associate propagated masklets ${\hat{\mathcal{M}}}_t$ with new object masks emerging in the current frame ${\mathcal{O}}_t$,

跟踪器与视频架构。 给定视频和提示 $P$，我们使用检测器与跟踪器（见图4），在整个视频中检测并跟踪与提示对应的目标。 在每一帧上，检测器发现新目标 ${\mathcal{O}}_t$，跟踪器则将前一时刻 $t-1$ 帧中的掩码序列 ${\mathcal{M}}_{t-1}$（时空掩码）传播到当前时刻 $t$ 帧上的新位置 ${\hat{\mathcal{M}}}_t$。 我们使用匹配函数，将传播得到的掩码序列 ${\hat{\mathcal{M}}}_t$ 与当前帧中新出现的目标掩码 ${\mathcal{O}}_t$ 关联起来：

$$\begin{array}{rl}{\hat{\mathcal{M}}}_t& =\mathrm{propagate}\left({\mathcal{M}}_{t-1}\right),\\ {\mathcal{O}}_t& =\mathrm{detect}(I_t,P),\\ {\mathcal{M}}_t& =\mathrm{match}\mathrm{\_}\mathrm{and}\mathrm{\_}\mathrm{update}({\hat{\mathcal{M}}}_t,{\mathcal{O}}_t).\end{array}$$

Tracking an Object with SAM 2 Style Propagation. A masklet is initialized for every object detected on the first frame. Then, on each subsequent frame, the tracker module predicts the new masklet locations ${\hat{\mathcal{M}}}_t$ of those already-tracked objects based on their previous locations ${\mathcal{M}}_{t-1}$ through a single-frame propagation step similar to the video object segmentation task in SAM 2. The tracker shares the same image/frame encoder (PE backbone) as the detector. After training the detector, we freeze PE and train the tracker as in SAM 2, including a prompt encoder, mask decoder, memory encoder, and a memory bank that encodes the object's appearance using features from the past frames and conditioning frames (frames where the object is first detected or user-prompted). The memory encoder is a transformer with self-attention across visual features on the current frame and cross-attention from the visual features to the spatial memory features in the memory bank. We describe details of our video approach in the appendix. During inference, we only retain frames where the object is confidently present in the memory bank. The mask decoder is a two-way transformer between the encoder hidden states and the output tokens. To handle ambiguity, we predict three output masks for every tracked object on each frame along with their confidence, and select the most confident output as the predicted mask on the current frame.

使用 SAM 2 式传播跟踪目标。 我们为第一帧上检测到的每个目标初始化一个掩码序列。 随后，在每个后续帧上，跟踪器模块通过类似 SAM 2 视频目标分割任务的单帧传播步骤，根据已跟踪目标的先前位置 ${\mathcal{M}}_{t-1}$ 预测其新的掩码序列位置 ${\hat{\mathcal{M}}}_t$。 跟踪器与检测器共享同一个图像/帧编码器（PE 骨干网络）。 训练检测器后，我们冻结 PE，并像 SAM 2 一样训练跟踪器，其中包括提示编码器、掩码解码器、记忆编码器和记忆库；记忆库使用过去帧和条件帧（首次检测到目标或用户提供提示的帧）的特征编码目标外观。 记忆编码器是一个 Transformer，它在当前帧的视觉特征之间执行自注意力，并从视觉特征到记忆库中的空间记忆特征执行交叉注意力。 视频方法的细节见附录。 推理期间，我们仅在记忆库中保留能够确信目标存在的帧。 掩码解码器是在编码器隐藏状态和输出 token 之间运行的双向 Transformer。 为处理歧义，我们在每一帧上为每个被跟踪目标预测三个输出掩码及其置信度，并选择置信度最高的输出作为当前帧的预测掩码。

Matching and Updating Based on Detections. After obtaining the tracked masks ${\hat{\mathcal{M}}}_t$, we match them with the current frame detections ${\mathcal{O}}_t$ through a simple IoU based matching function and add them to ${\mathcal{M}}_t$ on the current frame. We further spawn new masklets for all newly detected objects that are not matched. The merging might suffer from ambiguities, especially in crowded scenes. We address this with two temporal disambiguation strategies outlined next. First, we use temporal information in the form of a masklet detection score to measure how consistently a masklet is matched to a detection within a temporal window (based on the number of past frames where it was matched to a detection). If a masklet's detection score falls below a threshold, we suppress it. Second, we use the detector outputs to resolve specific failure modes of the tracker due to occlusions or distractors. We periodically re-prompt the tracker with high-confidence detection masks ${\mathcal{O}}_t$, replacing the tracker's own predictions ${\hat{\mathcal{M}}}_t$. This ensures that the memory bank has recent and reliable references (other than the tracker's own predictions).

基于检测结果进行匹配与更新。 得到跟踪掩码 ${\hat{\mathcal{M}}}_t$ 后，我们通过一个简单的基于 IoU 的匹配函数将其与当前帧检测结果 ${\mathcal{O}}_t$ 匹配，并把它们添加到当前帧的 ${\mathcal{M}}_t$ 中。 对于所有未匹配的新检测目标，我们还会生成新的掩码序列。 合并过程可能出现歧义，尤其是在拥挤场景中。 我们采用下面两种时间消歧策略来处理这一问题。 首先，我们以掩码序列检测分数的形式使用时间信息，衡量一个掩码序列在时间窗口内与检测结果匹配的一致程度（依据它在多少个过去帧中与检测结果匹配）。 如果掩码序列的检测分数低于阈值，我们将其抑制。 其次，我们使用检测器输出来解决由遮挡或干扰目标造成的特定跟踪器失效模式。 我们定期使用高置信度检测掩码 ${\mathcal{O}}_t$ 对跟踪器重新提示，以替换跟踪器自身的预测 ${\hat{\mathcal{M}}}_t$。 这确保记忆库拥有近期且可靠的参照，而不只是跟踪器自身的预测。

Instance Refinement with Visual Prompts. After obtaining the initial set of masks (or masklets), SAM 3 allows refining individual masks(lets) using positive and negative clicks. Specifically, given the user clicks, we apply the prompt encoder to encode them, and feed the encoded prompt into the mask decoder to predict an adjusted mask. In videos the mask is then propagated across the entire video to obtain a refined masklet. Training Stages. We train SAM 3 in four stages that progressively add data and capabilities: 1) Perception Encoder (PE) pre-training, 2) detector pre-training, 3) detector fine-tuning, and 4) tracker training with a frozen backbone. See the appendix for details.

使用视觉提示细化实例。 得到初始掩码（或掩码序列）集合后，SAM 3 允许使用正点击和负点击细化单个掩码（序列）。 具体而言，给定用户点击后，我们使用提示编码器对其编码，并将编码后的提示送入掩码解码器，以预测调整后的掩码。 在视频中，该掩码随后会传播到整个视频，从而得到细化的掩码序列。 训练阶段。 我们分四个阶段训练 SAM 3，逐步加入数据和能力：1）感知编码器（PE）预训练；2）检测器预训练；3）检测器微调；4）冻结骨干网络的跟踪器训练。 详细信息见附录。

4. Data Engine

图5：SAM 3 最终数据引擎概览。媒体与名词短语经过掩码生成、掩码验证、穷尽性验证和人工修正，形成训练数据。

Achieving a step change in PCS with SAM 3 requires training on a large, diverse set of concepts and visual domains, beyond existing datasets. We build an efficient data engine that iteratively generates annotated data via a feedback loop with SAM 3, human annotators, and AI annotators, actively mining media-phrase pairs on which the current version of SAM 3 fails to produce high-quality training data to further improve the model. By delegating certain tasks to AI annotators---models that match or surpass human accuracy---we more than double the throughput compared to a human-only annotation pipeline. We develop the data engine in four phases, with each phase increasing the use of AI models to steer human effort to the most challenging failure cases, alongside expanding visual domain coverage. Phases 1-3 focus only on images, with Phase 4 expanding to videos. We describe the key steps here; details and metrics are in the appendix.

要借助 SAM 3 实现 PCS 能力的跃升，需要在超出现有数据集范围的大规模、多样化概念和视觉领域集合上训练。 我们构建了一个高效数据引擎，通过 SAM 3、人类标注员和人工智能标注员组成的反馈循环迭代生成标注数据；它主动挖掘当前版本 SAM 3 无法生成高质量训练数据的媒体—短语对，以进一步改进模型。 将部分任务委托给准确率达到或超过人类的人工智能标注员后，与纯人工标注流水线相比，吞吐量提高了一倍以上。 我们分四个阶段开发数据引擎；每个阶段都会更多地使用人工智能模型，把人力引导到最具挑战性的失败案例上，同时扩大视觉领域覆盖范围。 第 1 至第 3 阶段仅关注图像，第 4 阶段扩展到视频。 这里介绍关键步骤，详细信息和指标见附录。

Data Engine Components (Figure 5). Media inputs (image or video) are mined from a large pool with the help of a curated ontology. An AI model proposes noun phrases (NPs) describing visual concepts, followed by another model (e.g., SAM 3) that generates candidate instance masks for each proposed NP. The proposed masks are verified by a two-step process: first, in Mask Verification (MV) annotators accept or reject masks based on their quality and relevance to the NP. Second, in Exhaustivity Verification (EV) annotators check if all instances of the NP have been masked in the input. Any media-NP pairs that did not pass the exhaustivity check are sent to a manual correction stage, where humans add, remove or edit masks (using SAM 1 in a browser based tool), or use “group” masks for small, hard to separate objects. Annotators may reject ungroundable or ambiguous phrases.

数据引擎组件（图5）。 在策展本体的帮助下，我们从一个大型媒体池中挖掘媒体输入（图像或视频）。 一个人工智能模型提出描述视觉概念的名词短语（NP），随后另一个模型（例如 SAM 3）为每个提出的 NP 生成候选实例掩码。 候选掩码通过两步流程验证：首先，在*掩码验证（MV）*中，标注员根据掩码质量及其与 NP 的相关性接受或拒绝掩码。 其次，在*穷尽性验证（EV）*中，标注员检查输入中该 NP 的所有实例是否都已被掩码覆盖。 任何未通过穷尽性检查的媒体—NP 对都会送入人工修正阶段；人类在基于浏览器的工具中使用 SAM 1 添加、移除或编辑掩码，或为尺寸很小且难以分开的目标使用“群组”掩码。 标注员可以拒绝无法定位或存在歧义的短语。

Phase 1: Human Verification. We first randomly sample images and NP proposal with a simple captioner and parser. The initial mask proposal model is SAM 2 prompted with the output of an off-the-shelf open-vocabulary detector, and initial verifiers are human. In this phase, we collected 4.3M image-NP pairs as the initial SA-Co/HQ dataset. We train SAM 3 on this data and use it as the mask proposal model for the next phase.

第 1 阶段：人工验证。 我们首先使用简单的图像描述器和解析器随机采样图像并提出 NP。 初始掩码候选模型是由现成开放词汇检测器的输出提示的 SAM 2，初始验证器则由人类担任。 在这一阶段，我们收集了 430 万个图像—NP 对，作为初始 SA-Co/HQ 数据集。 我们在这些数据上训练 SAM 3，并在下一阶段将其用作掩码候选模型。

Phase 2: Human + AI Verification. In this next phase, we use human accept/reject labels from the MV and EV tasks collected in Phase 1 to fine-tune Llama 3.2 to create AI verifiers that automatically perform the MV and EV tasks. These models receive image-phrase-mask triplets and output multiple-choice ratings of mask quality or exhaustivity. This new auto-verification process allows our human effort to be focused on the most challenging cases. We continue to re-train SAM 3 on newly collected data and update it 6 times. As SAM 3 and AI verifiers improve, a higher proportion of labels are auto-generated, further accelerating data collection. The introduction of AI verifiers for MV and EV roughly doubles the data engine's throughput vs. human annotators. We refer to the appendix for detailed analysis of how AI verifiers improve the data engine's throughput. We further upgrade the NP proposal step to a Llama-based pipeline that also proposes hard negative NPs adversarial to SAM 3. Phase 2 adds 122M image-NP pairs to SA-Co/HQ.

第 2 阶段：人类与人工智能验证。 在这一阶段，我们使用第 1 阶段 MV 和 EV 任务收集的人类接受/拒绝标签微调 Llama 3.2，创建可自动执行 MV 和 EV 任务的人工智能验证器。 这些模型接收图像—短语—掩码三元组，并以选择题形式输出对掩码质量或穷尽性的评分。 这一新的自动验证流程使我们能够把人力集中到最具挑战性的案例上。 我们继续使用新收集的数据重新训练 SAM 3，并对其更新 6 次。 随着 SAM 3 和人工智能验证器不断改进，自动生成标签的比例越来越高，进一步加快数据收集。 为 MV 和 EV 引入人工智能验证器后，数据引擎的吞吐量相较人类标注员大约提高一倍。 人工智能验证器如何提升数据引擎吞吐量的详细分析见附录。 我们还将 NP 提出步骤升级为基于 Llama 的流水线，该流水线还会提出对 SAM 3 具有对抗性的困难负 NP。 第 2 阶段向 SA-Co/HQ 增加了 1.22 亿个图像—NP 对。

图6：SA-Co 中带有短语与实例掩码/身份标注的视频（上）和图像（下）示例。

Phase 3: Scaling and Domain Expansion. In the third phase, we use AI models to mine increasingly challenging cases and broaden domain coverage in SA-Co/HQ to 15 datasets. A domain is a unique distribution of text and visual data. In new domains, the MV AI verifier performs well zero-shot, but the EV AI verifier needs to be improved with modest domain-specific human supervision. We also expand concept coverage to long-tail, fine-grained concepts by extracting NPs from the image alt-text where available and by mining concepts from a 22.4M node SA-Co ontology based on Wikidata (17 top-level categories, 72 sub-categories). We iterate SAM 3 training 7 times and AI verifiers 3 times, and add 19.5M image-NP pairs to SA-Co/HQ.

第 3 阶段：扩展规模与领域。 在第三阶段，我们使用人工智能模型挖掘难度不断增加的案例，并将 SA-Co/HQ 的领域覆盖扩展到 15 个数据集。 领域是文本数据和视觉数据的一种独特分布。 在新领域中，MV 人工智能验证器的零样本表现良好，但 EV 人工智能验证器需要少量特定领域的人工监督来改进。 我们还通过从可用的图像替代文本中提取 NP，并从基于 Wikidata、拥有 2240 万个节点的 SA-Co 本体（17 个顶级类别、72 个子类别）中挖掘概念，将概念覆盖扩展到长尾细粒度概念。 我们迭代训练 SAM 3 七次、人工智能验证器三次，并向 SA-Co/HQ 增加了 1950 万个图像—NP 对。

Phase 4: Video Annotation. This phase extends the data engine to video. We use a mature image SAM 3 to collect targeted quality annotations that capture video-specific challenges. The data mining pipeline applies scene/motion filters, content balancing, ranking, and targeted searches. Video frames are sampled (randomly or by object density) and sent to the image annotation flow (from phase 3). Masklets (spatio-temporal masks) are produced with SAM 3 (now extended to video) and post-processed via deduplication and removal of trivial masks. Because video annotation is more difficult, we concentrate humans on likely failures by favoring clips with many crowded objects and tracking failures. The collected video data SA-Co/VIDEO consists of 52.5K videos and 467K masklets. See the appendix for details.

第 4 阶段：视频标注。 这一阶段将数据引擎扩展到视频。 我们使用成熟的图像版 SAM 3 收集有针对性的高质量标注，以涵盖视频特有的挑战。 数据挖掘流水线应用场景/运动筛选、内容平衡、排序和定向搜索。 视频帧以随机方式或按目标密度采样，并送入第 3 阶段的图像标注流程。 SAM 3（此时已扩展到视频）生成掩码序列（时空掩码），随后通过去重和移除无关紧要的掩码进行后处理。 由于视频标注更困难，我们优先处理包含大量拥挤目标和跟踪失败的片段，将人力集中到可能失败的案例上。 收集到的视频数据 SA-Co/VIDEO 包含 5.25 万个视频和 46.7 万个掩码序列。 详细信息见附录。

5. Segment Anything with Concepts (SA-Co) Dataset

Training Data. We collect three image datasets for the PCS task: (i) SA-Co/HQ, the high-quality image data collected from the data engine in phases 1-4, (ii) SA-Co/SYN, a synthetic dataset of images labeled by a mature data engine (phase 3) without human involvement, and (iii) SA-Co/EXT, 15 external datasets that have instance mask annotations, enriched with hard negatives using our ontology pipeline. Notably in the SA-Co/HQ dataset we annotate 5.2M images and 4M unique NPs, making it the largest high-quality open-vocab segmentation dataset. We also annotate a video dataset, SA-Co/VIDEO, containing 52.5K videos and 24.8K unique NPs, forming 134K video-NP pairs. The videos on average have 84.1 frames at 6 fps. See the appendix for details including full statistics, comparison with existing datasets and the distribution of concepts.

训练数据。 我们为 PCS 任务收集了三个图像数据集：（i）SA-Co/HQ，即数据引擎在第 1 至第 4 阶段收集的高质量图像数据；（ii）SA-Co/SYN，即由成熟数据引擎（第 3 阶段）在无人参与的情况下标注图像得到的合成数据集；（iii）SA-Co/EXT，即 15 个带有实例掩码标注的外部数据集，并使用本体流水线补充困难负样本。 特别地，我们在 SA-Co/HQ 数据集中标注了 520 万张图像和 400 万个独特 NP，使其成为规模最大的高质量开放词汇分割数据集。 我们还标注了一个视频数据集 SA-Co/VIDEO，其中包含 5.25 万个视频和 2.48 万个独特 NP，形成 13.4 万个视频—NP 对。 这些视频以 6 fps 采样，平均包含 84.1 帧。 完整统计信息、与现有数据集的比较以及概念分布等详细信息见附录。

SA-Co Benchmark. The SA-Co evaluation benchmark has 207K unique phrases, 121K images and videos, and over 3M media-phrase pairs with hard negative labels to test open-vocabulary recognition. It has 4 splits: SA-Co/Gold has seven domains and each image-NP pair is annotated by three different annotators (used to measure human performance); SA-Co/Silver has ten domains and only one human annotation per image-NP pair; SA-Co/Bronze and SA-Co/Bio are nine existing datasets either with existing mask annotations or masks generated by using boxes as prompts to SAM 2. The SA-Co/VEval benchmark has three domains and one annotator per video-NP pair. See the appendix for dataset statistics and Figure 6 for example annotations.

SA-Co 基准。 SA-Co 评估基准拥有 20.7 万个独特短语、12.1 万张图像和视频，以及超过 300 万个带困难负标签、用于测试开放词汇识别的媒体—短语对。 它包含 4 个划分：SA-Co/Gold 有 7 个领域，每个图像—NP 对由三位不同标注员标注（用于衡量人类性能）；SA-Co/Silver 有 10 个领域，每个图像—NP 对只有一份人工标注；SA-Co/Bronze 和 SA-Co/Bio 由 9 个现有数据集构成，其中要么已有掩码标注，要么使用边界框提示 SAM 2 生成掩码。 SA-Co/VEval 基准包含 3 个领域，每个视频—NP 对由一位标注员标注。 数据集统计信息见附录，标注示例见图6。

Metrics. We aim to measure the usefulness of the model in downstream applications. Detection metrics such as average precision (AP) do not account for calibration, which means that models can be difficult to use in practice. To remedy this, we only evaluate predictions with confidence above 0.5, effectively introducing a threshold that mimics downstream usages and enforces good calibration. The PCS task can be naturally split into two sub-tasks, localization and classification. We evaluate localization using positive micro F1 (${\mathrm{pmF}}_1$) on positive media-phrase pairs with at least one ground-truth mask. Classification is measured with image-level Matthews Correlation Coefficient (IL_MCC) which ranges in $[-1,1]$ and evaluates binary prediction at the image level (“is the object present?”) without regard for mask quality. Our main metric, classification-gated F1 (cgF₁), combines these as follows:

指标。 我们旨在衡量模型在下游应用中的实用性。 平均精度（AP）等检测指标没有考虑校准问题，这意味着模型在实践中可能难以使用。 为解决这一问题，我们只评估置信度高于 0.5 的预测，相当于引入一个模拟下游用法并强制良好校准的阈值。 PCS 任务可以自然地拆分为定位和分类两个子任务。 我们在至少包含一个真实掩码的正媒体—短语对上，使用正样本微平均 F1（${\mathrm{pmF}}_1$）评估定位。 分类使用图像级 Matthews 相关系数（IL_MCC）衡量；该指标取值范围为 $[-1,1]$，在不考虑掩码质量的情况下评估图像级二元预测（“目标是否存在？”）。 我们的主要指标分类门控 F1（cgF₁）按如下方式组合二者：

$${\mathrm{cgF}}_1=100\times {\mathrm{pmF}}_1\times \mathrm{IL}\mathrm{\_}\mathrm{MCC}.$$

Full definitions are in the appendix. Handling Ambiguity. We collect 3 annotations per NP on SA-Co/Gold. We measure oracle accuracy comparing each prediction to all ground truths and selecting the best score. See the appendix.

完整定义见附录。 处理歧义。 我们在 SA-Co/Gold 上为每个 NP 收集 3 份标注。 我们将每个预测与所有真实标注比较并选择最高分，以衡量预言机准确率。 详细信息见附录。

6. Experiments

We evaluate SAM 3 across image and video segmentation, few-shot adaptation to detection and counting benchmarks, and segmentation with complex language queries with SAM 3 + MLLM. We also show a subset of ablations, with more in the appendix. References, more results and details are in the appendix.

我们在图像和视频分割、检测与计数基准上的少样本适应，以及使用 SAM 3 + MLLM 处理复杂语言查询的分割任务上评估 SAM 3。 我们还展示部分消融实验，更多结果见附录。 参考资料、更多结果和详细信息见附录。

表1：使用文本进行图像概念分割的评估。APₒ 对应 COCO-O 准确率；★ 表示部分使用 LVIS 训练；† 表示结果取自原论文；δ 表示结果取自 DINO-X API。灰色数值表示使用了相应的封闭集训练数据（LVIS/COCO）。

	Instance Segmentation						Box Detection								Semantic Segmentation
Model	LVIS		SA-Co				LVIS		COCO		SA-Co				ADE-847	PC-59	Cityscapes
	cgF1	AP	Gold	Silver	Bronze	Bio	cgF1	AP	AP	APo	Gold	Silver	Bronze	Bio	mIoU	mIoU	mIoU
			cgF1	cgF1	cgF1	pmF1					cgF1	cgF1	cgF1	pmF1
Human	--	--	72.8	--	--	--	--	--	--	--	74.0	--	--	--	--	--	--
OWLv2	20.1	--	17.3	7.6	3.9	0.64	19.9	35.2	38.2	42.4	16.9	7.1	4.1	0.95	--	--	--
OWLv2★	29.3	43.4	24.6	11.5	11.7	0.04	30.2	45.5	46.1	23.9	24.5	11.0	12.0	0.08	--	--	--
gDino-T	14.7	--	3.3	2.7	7.0	0.34	15.1	20.5	45.7	35.3	3.4	2.5	7.6	0.35	--	--	--
LLMDet-L	35.1	36.3	6.5	7.1	12.5	0.15	39.3	42.0	55.6	49.8	6.8	6.7	14.0	0.17	--	--	--
APE-D★	--	53.0†	16.4	7.3	12.4	0.00	--	59.6†	58.3†	--	17.3	7.7	14.3	0.00	9.2†	58.5†	44.2†
DINO-X	--	38.5†	21.3δ	--	--	--	--	52.4†	56.0†	--	22.5δ	--	--	--	--	--	--
Gemini 2.5	13.4	--	13.0	8.3	7.3	10.7	16.1	--	--	--	14.4	9.4	8.2	12.4	--	--	--
SAM 3	37.2	48.5	54.1	49.6	42.6	55.4	40.6	53.6	56.4	55.7	55.7	50.0	47.1	56.3	13.8	60.8	65.2

Image PCS with Text. We evaluate instance segmentation, box detection, and semantic segmentation on external and our benchmarks. SAM 3 is prompted with a single NP at a time, and predicts instance masks, bounding boxes, or semantic masks. As baselines, we evaluate OWLv2, GroundingDino (gDino), and LLMDet on box detection, and prompt SAM 1 with their boxes to evaluate segmentation. We also compare to APE, DINO-X, and Gemini 2.5 Flash, a generalist LLM. Table 1 shows that zero-shot, SAM 3 sets a new state-of-the-art on closed-vocabulary COCO, COCO-O and on LVIS boxes, and is significantly better on LVIS masks. On open-vocabulary SA-Co/Gold SAM 3 achieves more than double the cgF₁ score of the strongest baseline OWLv2^★, and 74% of the estimated human performance. The improvements are even higher on the other SA-Co splits. Open vocabulary semantic segmentation results on ADE-847, PascalConcept-59, and Cityscapes (val set) show that SAM 3 outperforms APE, a strong specialist baseline. See the appendix for details.

使用文本的图像 PCS。 我们在外部基准和自建基准上评估实例分割、边界框检测和语义分割。 每次使用一个 NP 提示 SAM 3，模型预测实例掩码、边界框或语义掩码。 作为基线，我们在边界框检测上评估 OWLv2、GroundingDino（gDino）和 LLMDet，并使用它们的边界框提示 SAM 1 来评估分割。 我们还与 APE、DINO-X 和通用 LLM Gemini 2.5 Flash 进行比较。 表1表明，在零样本设置下，SAM 3 在封闭词汇 COCO、COCO-O 和 LVIS 边界框上建立了新的最先进水平，并在 LVIS 掩码上显著更优。 在开放词汇 SA-Co/Gold 上，SAM 3 的 cgF₁ 分数是最强基线 OWLv2^★ 的两倍以上，并达到估计人类性能的 74%。 在其他 SA-Co 划分上的提升甚至更高。 ADE-847、PascalConcept-59 和 Cityscapes（验证集）上的开放词汇语义分割结果表明，SAM 3 优于强专业基线 APE。 详细信息见附录。

表2：在真实场景数据集上的零样本和 10 样本迁移结果。

Model	ODinW13		RF-100VL
	AP0	AP10	AP0	AP10
Gemini2.5-Pro	33.7	--	11.6	9.8
gDino-T	49.7	--	15.7	33.7
gDino1.5-Pro	58.7	67.9	--	--
SAM 3	61.0	71.8	15.2	36.5

表3：使用一个示例提示在 COCO、LVIS 和 ODinW13 上的结果。T、I 和 T+I 分别表示纯文本、纯图像及文本与图像组合提示；AP⁺ 仅在正样本上评估。

Model	COCO				LVIS				ODinW13
	AP	AP+	AP+	AP+	AP	AP+	AP+	AP+	AP	AP+	AP+	AP+
	T	T	I	T+I	T	T	I	T+I	T	T	I	T+I
T-Rex2	52.2	--	58.5	--	45.8	--	65.8	--	50.3	--	61.8	--
SAM 3	56.4	58.8	76.8	78.1	52.4	54.7	76.0	78.4	61.1	63.1	82.2	81.8

Few-Shot Adaptation. We evaluate zero- and few-shot transfer of SAM 3 on ODinW13 and RF100-VL, with their original labels as prompts. We do not perform any prompt tuning. We fine-tune SAM 3 without mask loss, and report average bbox mAP in Table 2. SAM 3 achieves state-of-the-art 10-shot performance, surpassing in-context prompting in Gemini and object detection experts (gDino); more details are in the appendix. RF-100VL contains domains with specialized prompts that are out of SAM 3's current scope, but SAM 3 adapts through fine-tuning more efficiently than baselines.

少样本适应。 我们使用原始标签作为提示，在 ODinW13 和 RF100-VL 上评估 SAM 3 的零样本和少样本迁移。 我们不进行任何提示调优。 我们在不使用掩码损失的情况下微调 SAM 3，并在表2中报告平均边界框 mAP。 SAM 3 取得最先进的 10 样本性能，超过 Gemini 的上下文内提示和目标检测专家（gDino）；更多细节见附录。 RF-100VL 包含带专业提示的领域，超出了 SAM 3 当前范围，但 SAM 3 通过微调进行适应的效率高于基线。

图7：在 SA-Co/Gold 短语上取平均，比较 SAM 3 与理想 PVS 基线的 cgF₁ 随交互式边界框提示数量的变化。

PCS with 1 Exemplar. We first evaluate image exemplars using a single input box sampled at random from the ground truth. This can be done only on “positive” data, where each prompted object appears in the image. We report the corresponding ${\mathbf{AP}}^{+}$ in Table 3 across three settings: text prompt (T), exemplar image (I), and both text and image (T+I); SAM 3 outperforms prior state-of-the-art T-Rex2 by a healthy margin on COCO (+18.3), LVIS (+10.3), and ODinW (+20.5). See the appendix for more details and results on SA-Co/Gold.

使用 1 个示例的 PCS。 我们首先使用从真实标注中随机采样的单个输入框评估图像示例。 这只能在每个被提示目标都出现在图像中的“正”数据上完成。 我们在表3中报告文本提示（T）、示例图像（I）以及文本与图像（T+I）三种设置下相应的 ${\mathbf{AP}}^{+}$；SAM 3 在 COCO（+18.3）、LVIS（+10.3）和 ODinW（+20.5）上均以明显优势超过此前最先进的 T-Rex2。 更多细节和 SA-Co/Gold 上的结果见附录。

PCS with K Exemplars. Next, we evaluate SAM 3 in an interactive setting, simulating collaboration with a human annotator. Starting with a text prompt, we iteratively add one exemplar prompt at a time: missed ground truths are candidate positive prompts, false positive detections are candidate negative prompts. Results (Figure 7) are compared to a perfect PVS baseline, where we simulate the user manually fixing errors using ideal box-to-mask corrections. SAM 3's PCS improves cgF₁ more quickly, as it generalizes from exemplars (e.g., detecting or suppressing similar objects), while PVS only corrects individual instances. After 3 clicks, interactive PCS outperforms text-only by +21.6 cgF₁ points and PVS refinement by +2.0. Performance plateaus after 4 clicks, as exemplars cannot fix poor-quality masks. Simulating a hybrid switch to PVS at this point yields gains, showing complementary.

使用 K 个示例的 PCS。 接下来，我们在交互设置下评估 SAM 3，模拟它与人类标注员协作。 从一个文本提示开始，我们每次迭代添加一个示例提示：漏检的真实目标作为候选正提示，假阳性检测作为候选负提示。 我们将结果（图7）与完美 PVS 基线比较；该基线模拟用户使用理想的边界框到掩码修正来手动修复错误。 SAM 3 的 PCS 能够从示例中泛化（例如检测或抑制相似目标），因此 cgF₁ 提升得更快，而 PVS 只能修正单个实例。 点击 3 次后，交互式 PCS 的 cgF₁ 比纯文本高 21.6 分，比 PVS 细化高 2.0 分。 点击 4 次后性能趋于稳定，因为示例无法修复低质量掩码。 此时模拟混合式切换到 PVS 可以获得增益，表明两者具有互补性。

表4：计数基准上的准确率。灰色表示使用了训练集。

Model	CountBench		PixMo-Count
	MAE ↓	Acc ↑	MAE ↓	Acc ↑
DINO-X	0.62	82.9	0.21	85.0
Qwen2-VL-72B	0.28	86.7	0.61	63.7
Molmo-72B	0.27	92.4	0.17	88.8
Gemini 2.5 Pro	0.24	92.4	0.38	78.2
SAM 3	0.12	93.8	0.21	86.2

Object Counting. We evaluate on object counting benchmarks CountBench and PixMo-Count to compare with several MLLMs using Accuracy (%) and Mean Absolute Error (MAE) from previous technical reports and our own evaluations. See Table 4 for results and the appendix for more evaluation details. Compared to MLLMs, SAM 3 not only achieves good object counting accuracy, but also provides object segmentation that most MLLMs cannot provide.

目标计数。 我们在目标计数基准 CountBench 和 PixMo-Count 上进行评估，并使用先前技术报告和自行评估得到的准确率（%）及平均绝对误差（MAE），与多个 MLLM 比较。 结果见表4，更多评估细节见附录。 与 MLLM 相比，SAM 3 不仅取得了良好的目标计数准确率，还能提供大多数 MLLM 无法提供的目标分割。

表5：使用文本提示在 SA-Co/VEval 与公开基准上进行视频 PCS（开放词汇视频实例分割）。SAM 3 尤其在包含大量 NP 的基准上表现强劲。†：GLEE 和 LLMDet 在 SA-Co/VEval 上的零样本表现不佳。

	SA-Co/VEval benchmark test split						Public benchmarks
Model	SA-V		YT-Temporal-1B		SmartGlasses		LVVIS	BURST	YTVIS21	OVIS
	(2.0K NPs)		(1.7K NPs)		(2.4K NPs)		(1.2K NPs)	(482 NPs)	(40 NPs)	(25 NPs)
	cgF1	pHOTA	cgF1	pHOTA	cgF1	pHOTA	test mAP	test HOTA	val mAP	val mAP
Human	53.1	70.5	71.2	78.4	58.5	72.3	--	--	--	--
GLEE† (all NPs at once)	0.1	8.7	1.6	16.7	0.0	4.7	20.8	28.4	62.2	38.7
GLEE† (one NP at a time)	0.1	11.8	2.2	18.9	0.1	5.6	9.3	20.2	56.5	32.4
LLMDet† + SAM 3 Tracker	2.3	30.1	8.0	37.9	0.3	18.6	15.2	33.3	31.3	20.4
SAM 3 Detector + T-by-D	25.7	55.7	47.6	68.2	29.7	60.0	35.9	39.7	56.5	55.1
SAM 3	30.3	58.0	50.8	69.9	36.4	63.6	36.3	44.5	57.4	60.5

Video PCS with Text. We evaluate video segmentation with text prompts on both our SA-Co/VEval benchmark and existing public benchmarks. For SA-Co/VEval, we report cgF₁ and pHOTA metrics across its subsets (SA-V, YT-Temporal-1B, SmartGlasses). For public benchmarks, we use their official metrics. Baselines include GLEE, an open-vocabulary image and video segmentation model, “LLMDet + SAM 3 Tracker” (replacing our detector with LLMDet), and “SAM 3 Detector + T-by-D” (replacing our tracker with an association module based on the tracking-by-detection paradigm). In Table 5, SAM 3 largely outperforms these baselines, especially on benchmarks with a very large number of noun phrases. On SA-Co/VEval it reaches over 80% of human pHOTA. See the appendix for more details.

使用文本的视频 PCS。 我们在自建 SA-Co/VEval 基准和现有公开基准上使用文本提示评估视频分割。 对于 SA-Co/VEval，我们在其各子集（SA-V、YT-Temporal-1B、SmartGlasses）上报告 cgF₁ 和 pHOTA 指标。 对于公开基准，我们使用其官方指标。 基线包括开放词汇图像和视频分割模型 GLEE、“LLMDet + SAM 3 Tracker”（用 LLMDet 替换我们的检测器），以及“SAM 3 Detector + T-by-D”（用基于按检测跟踪范式的关联模块替换我们的跟踪器）。 在表5中，SAM 3 大幅超过这些基线，尤其是在包含大量名词短语的基准上。 在 SA-Co/VEval 上，它达到人类 pHOTA 的 80% 以上。 更多细节见附录。

表6：SAM 3 相较 SAM 2 改进了视频目标分割（VOS）。† 表示零样本结果。

	J&F					G	J&Ḟ
Model	MOSEv1 val	DAVIS17 val	LVOSv2 val	SA-V val	SA-V test	YTVOS19 val	MOSEv2 val
SAMURAI	72.6	89.9	84.2	79.8	80.0	88.3	51.1
SAM2Long	75.2	91.4	85.9	81.1	81.2	88.7	51.5
SeC	75.3	91.3	86.5	82.7	81.7	88.6	53.8
SAM 2.1 L	77.9	90.7	79.6	77.9	78.4	89.3	47.9†
SAM 3	78.4	92.2	88.5	83.5	84.4	89.7	60.3

表7：SA-37 基准上的交互式图像分割结果。

Model	1-click Avg. mIoU	3-clicks Avg. mIoU	5-clicks Avg. mIoU	FPS
SAM 1 H	58.5	77.0	82.1	41.0
SAM 2.1 L	66.4	80.3	84.3	93.0
SAM 3	66.1	81.3	85.1	43.5

PVS. We evaluate SAM 3 on a range of visual prompting tasks, including Video Object Segmentation (VOS) and interactive image segmentation. Table 6 compares SAM 3 to recent state-of-the-art methods on the VOS task. SAM 3 achieves significant improvements over SAM 2 on most benchmarks, particularly on the challenging MOSEv2 dataset, where SAM 3 outperforms prior work by 6.5 points. For the interactive image segmentation task, we evaluate SAM 3 on the 37 datasets benchmark introduced in SAM 2. As shown in Table 7, SAM 3 outperforms SAM 2 on average $\mathrm{mIoU}$. See also the appendix for interactive video segmentation.

PVS。 我们在一系列视觉提示任务上评估 SAM 3，包括视频目标分割（VOS）和交互式图像分割。 表6在 VOS 任务上将 SAM 3 与近期最先进方法进行比较。 SAM 3 在大多数基准上都相较 SAM 2 取得显著提升，特别是在具有挑战性的 MOSEv2 数据集上，它比此前工作高 6.5 分。 对于交互式图像分割任务，我们在 SAM 2 提出的 37 数据集基准上评估 SAM 3。 如表7所示，SAM 3 的平均 $\mathrm{mIoU}$ 优于 SAM 2。 交互式视频分割结果另见附录。

表8：SAM 3 Agent 结果。灰色表示在 ReasonSeg 训练集上微调的结果，* 表示复现结果，下划线表示主要指标。†：ReasonSeg 使用 LISA-13B-LLaVA1.5，OmniLabel 使用 REAL。

Model	MLLM	ReasonSeg (gIoU)				OmniLabel (AP)
		val	test			val 2023
		All	All	Short	Long	descr	descr-S	descr-M	descr-L
X-SAM	Phi-3-3.8B	56.6	57.8	47.7	56.0	12.0*	17.1*	11.4*	8.8*
SegZero	Qwen2.5-VL 7B	62.6	57.5	--	--	13.5*	20.7*	12.4*	9.1*
RSVP	GPT-4o	64.7	55.4	61.9	60.3	--	--	--	--
Overall state-of-the-art†		65.0	61.3	55.4	63.2	36.5	54.4	33.2	25.5
SAM 3 Agent	Qwen2.5-VL 7B	62.2	63.0	59.4	64.1	36.7	52.6	34.3	26.6
SAM 3 Agent	Llama4 Maverick	68.5	67.1	66.8	67.2	32.8	43.7	30.9	27.5
SAM 3 Agent	Qwen2.5-VL 72B	74.6	70.8	70.3	71.0	42.0	56.0	40.4	33.2
SAM 3 Agent	Gemini 2.5 Pro	77.0	74.0	75.8	73.4	45.3	53.8	45.1	37.7

SAM 3 Agent. We experiment with an MLLM that uses SAM 3 as a tool to segment more complex text queries. The MLLM proposes noun phrase queries to prompt SAM 3 and analyzes the returned masks, iterating until the masks are satisfactory. Table 8 shows that this “SAM 3 Agent” evaluated zero-shot on ReasonSeg and OmniLabel surpasses prior work without training on any referring expression segmentation or reasoning segmentation data. SAM 3 Agent also outperforms previous zero-shot results on RefCOCO+ and RefCOCOg. SAM 3 can be combined with various MLLMs, with the same set of the system prompts for all those MLLMs, showing SAM 3's robustness. See the appendix for more details.

SAM 3 Agent。 我们试验了一个将 SAM 3 用作工具来分割更复杂文本查询的 MLLM。 该 MLLM 提出名词短语查询来提示 SAM 3，并分析返回的掩码，反复迭代直到掩码令人满意。 表8表明，在 ReasonSeg 和 OmniLabel 上零样本评估的“SAM 3 Agent”无需使用任何指代表达式分割或推理分割数据训练，就能超过此前工作。 SAM 3 Agent 也超过了 RefCOCO+ 和 RefCOCOg 上此前的零样本结果。 SAM 3 可以与多种 MLLM 结合，并且所有这些 MLLM 使用同一组系统提示，体现了 SAM 3 的鲁棒性。 更多细节见附录。

表9：SA-Co/Gold 上选取的模型与数据消融。不同子表之间的数值不可直接比较。

(a) Presence head

Use	cgF1	IL_MCC	pmF1
×	50.7	0.77	65.4
✓	52.2	0.82	63.4

(b) Hard Negatives

#/img	cgF1	IL_MCC	pmF1
0	28.3	0.44	62.4
5	39.4	0.62	62.9
15	41.8	0.67	62.4
30	43.0	0.68	62.8

(c) Training data

EXT	SYN	HQ	cgF1	IL_MCC	pmF1
✓	×	×	23.7	0.46	50.4
✓	✓	×	32.8	0.57	56.9
✓	×	✓	45.5	0.71	64.0
✓	✓	✓	47.4	0.74	63.8

(d) SAM 3 + AI verifiers

Model	cgF1	IL_MCC	pmF1
Human	72.8	0.94	77.0
SAM 3	54.0	0.82	65.9
+ EV AI	61.2	0.86	70.8
+ MV AI	62.3	0.87	71.1

Selected Ablations. In Table 9 we report a subset of the more extensive ablations from the appendix. Note that the ablated models are from different, shorter training runs than the model evaluated above. The presence head boosts cgF₁ by +1.5, improving image-level recognition measured by IL_MCC by +0.05. Adding hard negatives significantly improves the model performance, most notably the image-level IL_MCC from 0.44 to 0.68. Synthetic (SYN) training data improves over the external (EXT) by +8.8 cgF₁ and our high-quality (HQ) annotations add +14.6 cgF₁ on top of this baseline. We present detailed data scaling laws of both types of data in the appendix, showing their effectiveness on both in-domain and out-of-domain test sets. We also show how AI verifiers can improve pseudo-labels. Replacing the presence score from SAM 3 with that score from the exhaustivity verification (EV) AI verifier boosts cgF₁ by +7.2. Using the mask verification (MV) AI verifier to remove bad masks adds another 1.1 points. Overall, AI verifiers close half of the gap between SAM 3's and human performance.

选取的消融实验。 我们在表9中报告附录中更大规模消融实验的一个子集。 请注意，被消融的模型来自与上述评估模型不同且更短的训练过程。 存在性头使 cgF₁ 提高 1.5，并使 IL_MCC 衡量的图像级识别提高 0.05。 加入困难负样本显著提高模型性能，其中最明显的是图像级 IL_MCC 从 0.44 提高到 0.68。 合成（SYN）训练数据相较外部（EXT）数据使 cgF₁ 提高 8.8，而我们的高质量（HQ）标注在此基线上又提高 14.6 cgF₁。 我们在附录中给出两类数据的详细数据缩放规律，表明它们对域内和域外测试集均有效。 我们还展示了人工智能验证器如何改进伪标签。 将 SAM 3 的存在性分数替换为穷尽性验证（EV）人工智能验证器的分数，可使 cgF₁ 提高 7.2。 使用掩码验证（MV）人工智能验证器移除不良掩码，又增加了 1.1 分。 总体而言，人工智能验证器弥合了 SAM 3 与人类性能差距的一半。

图8：使用合成数据进行领域适应。SAM 3 与人工智能验证器（教师系统）生成的合成（SYN）数据，表现出与人工标注（HQ）数据相似的缩放行为。

Domain adaptation ablation. With domain-specific synthetic data generated by SAM 3 + AI verifiers, we show that one can significantly improve performance on a new domain without any human annotation. We hold out one of the SA-Co domains, “Food&drink”, from training SAM 3 and AI verifiers. We then use three variants of training data for the novel “Food&drink” domain: high-quality AI+human annotations as in SA-Co/HQ (referred to as SA-Co/HQ-Food), synthetic annotations as in SA-Co/SYN, using AI but no humans (SA-Co/SYN-Food), and pseudo-labels generated before the AI verification step, i.e. skipping both AI verifiers and humans (PL-Food). Figure 8 plots performance on the “Food&drink” test set of the SA-Co/Gold benchmark as each type of training data is scaled up. We mix the domain specific data and high-quality general domain data at a 1:1 ratio. PL-Food provides some improvement compared to the baseline SAM 3 (zero-shot), but is far below the other variants due to its lower quality. HQ-Food and SYN-Food show similar scaling behavior, with SYN-Food slightly lower but eventually catching up, without incurring any human annotation cost. This points to a scalable way to improve performance on new data distributions. More details are in the appendix.

领域适应消融。 借助 SAM 3 + 人工智能验证器生成的特定领域合成数据，我们表明可以在没有任何人工标注的情况下显著提高新领域上的性能。 训练 SAM 3 和人工智能验证器时，我们留出 SA-Co 的一个领域“Food&drink”。 随后，我们为这个新的“Food&drink”领域使用三种训练数据：类似 SA-Co/HQ 的高质量人工智能+人工标注（称为 SA-Co/HQ-Food）；类似 SA-Co/SYN、使用人工智能但不使用人类的合成标注（SA-Co/SYN-Food）；以及在人工智能验证步骤之前生成的伪标签，即同时跳过人工智能验证器和人类（PL-Food）。 图8绘制了随着各类训练数据规模增长，模型在 SA-Co/Gold 基准“Food&drink”测试集上的性能。 我们以 1:1 的比例混合特定领域数据和高质量通用领域数据。 与基线 SAM 3（零样本）相比，PL-Food 带来一定提升，但由于质量较低，远逊于其他变体。 HQ-Food 和 SYN-Food 表现出相似的缩放行为；SYN-Food 略低，但最终能够追上，而且不产生任何人工标注成本。 这为改进新数据分布上的性能指明了一条可扩展路径。 更多细节见附录。

7. Related Work

Promptable and Interactive Visual Segmentation. SAM introduces “promptable” image segmentation with interactive refinement. While the original task definition included text prompts, they were not fully developed. SAM 2 extended the promptable visual segmentation task to video, allowing refinement points on any frame. SAM 3 inherits geometry-based segmentation while extending to include text and image exemplar prompts to segment all instances of a concept in images and videos.

可提示与交互式视觉分割。 SAM 引入了带交互式细化的“可提示”图像分割。 尽管原始任务定义包括文本提示，但该能力并未得到充分发展。 SAM 2 将可提示视觉分割任务扩展到视频，允许在任意帧上提供细化点。 SAM 3 继承基于几何信息的分割，同时扩展到文本和图像示例提示，以分割图像和视频中某个概念的所有实例。

Open-Vocabulary Detection and Segmentation in Images exhaustively labels every instance of an open-vocabulary object category with a coarse bounding box (detection) or a fine-grained pixel mask (segmentation). Recent open-vocabulary (OV) detection and segmentation methods leverage large-scale vision-language encoders such as CLIP to handle categories described by arbitrary text, even those never seen during training. While DETR is limited to a closed set of categories seen during training, MDETR evolves the approach to condition on raw text queries. Image exemplars used as prompts to specify the desired object category (e.g., DINOv, T-Rex2) present a practical alternative to text, but fall short in conveying the abstract concept of objects as effectively as text prompts. We introduce a new benchmark for OV segmentation with $>100\times$ more unique concepts than prior work.

图像中的开放词汇检测与分割使用粗粒度边界框（检测）或细粒度像素掩码（分割），穷尽标注某个开放词汇目标类别的每个实例。 近期开放词汇（OV）检测与分割方法利用 CLIP 等大规模视觉—语言编码器处理由任意文本描述的类别，包括训练期间从未见过的类别。 DETR 仅限于训练期间见过的封闭类别集合，而 MDETR 将该方法发展为以原始文本查询作为条件。 使用图像示例作为提示来指定所需目标类别（例如 DINOv、T-Rex2）是文本的一种实用替代方案，但在表达目标抽象概念方面不如文本提示有效。 我们引入一个新的 OV 分割基准，其独特概念数量是先前工作的 $>100\times$。

Visual Grounding localizes a language expression referring to a region of the image with a box or mask. Prior work introduces phrase detection as both deciding whether the phrase is relevant to an image and localizing it. GLIP and GroundingDino formulate object detection as phrase grounding, unifying both tasks during training. MQ-GLIP adds image exemplars to text as queries. Building on this trend toward models supporting multiple tasks and modalities, GLEE allows text phrases, referring expressions, and visual prompts for category and instance grounding in both images and videos. Unlike SAM 3, GLEE does not support exemplars or interactive refinement. LISA allows segmentation that requires reasoning, while OMG-LLaVa and GLaMM generate natural language responses interleaved with corresponding segmentation masks, with GLaMM accepting both textual and optional image prompts as input. Some general-purpose MLLMs can output boxes and masks (Gemini2.5) or points (Molmo). SAM 3 can be used as a “vision tool” in combination with an MLLM.

视觉定位使用边界框或掩码定位指向图像区域的语言表达式。 先前工作将短语检测定义为同时判断短语是否与图像相关并对其进行定位。 GLIP 和 GroundingDino 将目标检测形式化为短语定位，从而在训练期间统一这两个任务。 MQ-GLIP 将图像示例加入文本，作为查询。 顺应模型支持多任务和多模态的趋势，GLEE 允许在图像和视频中使用文本短语、指代表达式和视觉提示进行类别及实例定位。 与 SAM 3 不同，GLEE 不支持示例或交互式细化。 LISA 支持需要推理的分割，而 OMG-LLaVa 和 GLaMM 生成与相应分割掩码交错的自然语言响应，其中 GLaMM 同时接受文本提示和可选图像提示作为输入。 一些通用 MLLM 可以输出边界框和掩码（Gemini2.5）或点（Molmo）。 SAM 3 可以与 MLLM 结合，作为“视觉工具”使用。

Multi-Object Tracking and Segmentation methods identify object instances in video and track them, associating each with a unique ID. In tracking-by-detection methods, detection is performed independently on each frame to produce boxes and confidence scores, followed by association of boxes using motion-based and appearance-based matching as in SORT, Tracktor, ByteTrack, SAM2MOT, or OC-SORT. An alternative is an end-to-end trainable architecture that jointly detects and associates objects, e.g., TrackFormer, TransTrack, or MOTR. TrackFormer uses a DETR-like encoder-decoder that initializes new tracks from static object queries and auto-regressively follows existing tracks with identity-preserving track queries. A challenge with joint models is the conflict between detection and tracking, where one needs to focus on semantics while the other on disentangling identities, even if their spatial locations overlap over time. SAM 3 is a strong image detector tightly integrated into a tracker to segment concepts in videos.

多目标跟踪与分割方法识别视频中的目标实例并对其进行跟踪，为每个实例关联唯一 ID。 在按检测跟踪方法中，每一帧都独立执行检测以生成边界框和置信度分数，随后像 SORT、Tracktor、ByteTrack、SAM2MOT 或 OC-SORT 一样，使用基于运动和外观的匹配来关联边界框。 另一种选择是可端到端训练、同时检测和关联目标的架构，例如 TrackFormer、TransTrack 或 MOTR。 TrackFormer 使用类似 DETR 的编码器—解码器，通过静态目标查询初始化新轨迹，并使用保持身份的轨迹查询自回归地跟踪现有轨迹。 联合模型面临的一项挑战是检测与跟踪之间的冲突：前者需要关注语义，后者需要区分身份，即使它们的空间位置会随时间重叠。 SAM 3 是一个与跟踪器紧密集成的强大图像检测器，用于分割视频中的概念。

8. Conclusion

We present Segment Anything with Concepts, enabling open-vocabulary text and image exemplars as prompts in interactive segmentation. Our principal contributions are: (i) introducing the PCS task and SA-Co benchmark, (ii) an architecture that decouples recognition, localization and tracking and extends SAM 2 to solve concept segmentation while retaining visual segmentation capabilities, (iii) a high-quality, efficient data engine that leverages the complimentary strengths of human and AI annotators. SAM 3 achieves state-of-the-art results, doubling performance over prior systems for PCS on SA-Co in images and videos. That said, our model has several limitations. For example, it struggles to generalize to out-of-domain terms, which could be mitigated by automatic domain expansion but requires extra training. We discuss this and other limitations of our model in the appendix. We believe SAM 3 and the SA-Co benchmark will be important milestones and pave the way for future research and applications in computer vision.

我们提出 Segment Anything with Concepts，使开放词汇文本和图像示例能够作为交互式分割中的提示。 我们的主要贡献包括：（i）引入 PCS 任务和 SA-Co 基准；（ii）提出一种解耦识别、定位和跟踪的架构，在保留视觉分割能力的同时扩展 SAM 2 以解决概念分割；（iii）构建高质量、高效率的数据引擎，利用人类标注员和人工智能标注员各自的互补优势。 SAM 3 取得了最先进结果，在图像和视频 SA-Co 上的 PCS 性能达到先前系统的两倍。 不过，我们的模型仍存在一些局限。 例如，它难以泛化到域外术语；自动领域扩展可以缓解这一问题，但需要额外训练。 我们在附录中讨论了这一问题以及模型的其他局限。 我们相信，SAM 3 和 SA-Co 基准将成为重要里程碑，并为计算机视觉未来的研究和应用铺平道路。