Optimizing multi-task network with learned prototypes for weakly supervised semantic segmentation

Abstract

Weakly supervised semantic segmentation (WSSS) presents a challenging task wherein semantic objects are extracted solely through the utilization of image-level labels as supervision. One common category of state-of-the-art solutions depends on the generation of pseudo pixel-level annotations via the use of localization maps. Nevertheless, in the majority of such solutions, the quality of pseudo annotations may not effectively fulfill the requirements of semantic segmentation owing to the incomplete nature of the localization maps. In order to generate denser localization maps for WSSS, this paper proposes the use of a prototype learning guided multi-task network. Initially, the prototypes (also referred to as prototypical feature vectors) are employed to depict the similarities between images. Specifically, the shared information among different training images is thoroughly exploited to concomitantly learn the prototypes for both foreground categories and background. This approach facilitates the localization of more reliable background pixels and foreground regions by evaluating the similarities between the representative prototypes and the extracted features of pixels. Additionally, the learned prototypes can be incorporated into the multi-task network to enhance the efficiency of parameter optimization by adaptively rectifying errors in pixel-level supervision. Therefore, the optimization of the multi-task network for object localization and the production of high-quality proxy annotations can be achieved by means of clean image-level labels and refined pixel-level supervision working in conjunction. By selecting and refining proxy annotations, the performance of the segmentation algorithm can be further improved. Extensive experiments conducted on two datasets, namely, PASCAL VOC 2012 and COCO 2014, have substantiated the fact that the prototype learning guided multi-task network being proposed outperforms the current state-of-the-art (SOTA) methods in terms of segmentation performance, achieving a mean IoU of 72.1% and 72.6% on the PASCAL VOC 2012 validation and test sets, respectively.