Self-supervised video object segmentation via pseudo label rectification

Abstract

In this paper we propose a novel self-supervised framework for video object segmentation (VOS) which consists of siamese encoders and bi-decoders. Siamese encoders extract multi-level features and generate pseudo labels for each pixel by cross attention in visual-semantic space. Such siamese encoders are learned via the colorization task without any labeled video data. Bi-decoders take in features from different layers of the encoder and output refined segmentation masks. Such bi-decoders are trained by the pseudo labels, and in turn pseudo labels are rectified via bi-decoders mutual learning. The variation of the bi-decoders’ outputs is minimized such that the gap between pseudo labels and the ground-truth is reduced. Experimental results on the challenging datasets DAVIS-2017 and YouTube-VOS demonstrate the effectiveness of our proposed approach.