LayerCLIP: A fine-grained class activation map for weakly supervised semantic segmentation

Abstract

Weakly supervised semantic segmentation (WSSS) using image-level labels aims to create pseudo-labels leveraging Class Activation Maps (CAM) to train a separate segmentation model. Recent methods that utilize Contrastive Language-Image Pre-training (CLIP) models have achieved significant advancements. These approaches take advantage of CLIP’s capability to identify various categories without requiring additional training. However, due to the limited local information of the final embedding layer, the CAM generated by the CLIP model is still a rough region with an under-activated or over-activated issue. Furthermore, the abundant multi-layer information of CLIP, which plays a vital role in dense prediction, has been ignored. In this paper, we proposed a LayerCLIP model for a fine-grained CAM generation via hierarchical features, which consists of two consecutive components: a dynamic hierarchical CAMs module and an adaptive affinity module. Specifically, the dynamic hierarchical CAMs module utilizes the hierarchical features to produce two complementary CAMs, along with a dynamic strategy to fuse these CAMs. Subsequently, the affinity based on multi-head self-attention is adaptively reweighted to refine CAM by the CAM itself in the adaptive affinity module. LayerCLIP significantly enhances the quality of CAM. Our method achieves a new state-of-the-art performance on PASCAL VOC 2012 (75.1 % mIoU) and MS COCO 2014 (46.9 % mIoU) through extensive benchmark experiments.