ADFNet: Attention-based Fusion Network for Few-shot RGB-D Semantic Segmentation

Abstract

∗Deep CNNs have made great progress in image semantic segmentation. However, they require a large-scale labeled image dataset, which might be costly. Moreover, the model can hardly generalize to unseen classes. Few-shot segmentation, which can learn to perform segmentation on new classes from a few labeled samples, has been developed recently to tackle the problem. In this paper, we proposed a novel prototype network to undertake the challenging task of few-shot semantic segmentation on complex scenes with RGB-D datasets, which is named ADFNet (Attention-based Depth Fusion Network). Our ADFNet learns class-specific prototypes from both RGB channels and depth channels. Meanwhile, we proposed an attention-based fusion module to fuse the depth feature into the image feature that can better utilize the information of the support depth images. We also proposed RELIEF-prototype which refines the prototype and provides an additional improvement to the model. Furthermore, we proposed a new few-shot RGB-D segmentation benchmark based on SUN RGB-D, named SUN RGB-D-5i. Experiments on SUN RGB-D-5i show that our method achieves the mIoU score of 27.4% and 34.6% for 1-shot and 5-shot settings respectively, outperforming the baseline method by 4.2% and 4.4% respectively.