From superpixels to foundational models: An overview of unsupervised and generalizable image segmentation

Image segmentation is one of the most classical computer vision tasks. Segmentation tasks yield a set of classes attributed to individual pixels instead of sparsely predicted images or patches, such as in classification or detection tasks. However, creating annotation sets for pixelwise tasks is a very costly task, often requiring hours for labeling single samples in images with multiple classes of objects. In this context, unsupervised learning can be leveraged either to expedite the annotation procedure and/or to guide the segmentation algorithms altogether without the need for manual annotations. Classical unsupervised segmentation methods leveraged techniques from areas as graph theory, image processing, clustering or supervised classifiers in order to achieve “shallow” pixelwise classification. These techniques usually aim to achieve superpixel over-segmentations by grouping similar pixels that should pertain to the same object. Modern deep unsupervised approaches for image segmentation aimed to group pixels in a data-driven way by using the capabilities of deep architectures to process unstructured data such as images. Later, self-supervised learning bypassed the need for labels via pretext tasks, compelling deep architectures to learn more generic features capable of enhancing downstream tasks, including segmentation. The generalized representations produced by unsupervised models have propelled the recent progress in self-supervised, few- and zero-shot learning and even general-purpose foundational models in computer vision, yielding state-of-the-art results across diverse tasks and datasets. This paper provides an overview of unsupervised and generalizable approaches for image segmentation, introduces key concepts and terminology, and discusses the main aspects of state-of-the-art methods. Additionally, we highlight prominent applications in various domains such as remote sensing, medical imaging, and geology. Finally, we discuss trends and future directions for state-of-the-art unsupervised image segmentation.