AESeg: Affinity-enhanced segmenter using feature class mapping knowledge distillation for efficient RGB-D semantic segmentation of indoor scenes

Abstract

Recent advances in deep learning for semantic segmentation models have introduced dynamic segmentation methods as opposed to static segmentation methods represented by full convolutional networks. Dynamic prediction methods replace static classifiers with learnable class embeddings to achieve global semantic awareness. Although dynamic methods excel in accuracy, the learning and inference of class embeddings is usually accompanied by a tedious computational burden. To address this challenge, we propose an affinity-enhanced semantic segmentation framework that synergistically combines the strengths of static and dynamic methodologies. Specifically, our approach leverages semantic features to obtain preliminary static segmentation results and constructs a binary affinity matrix that explicitly encodes pixel-wise category relationships. This affinity matrix serves as a dynamic classification kernel, effectively integrating global context awareness with static features, achieving comparable performance to purely dynamic approaches but with a substantially reduced computational overhead. Furthermore, we introduce a novel feature-to-category mapping refinement technique. This technique performs feature knowledge migration by learning a linear transformation between the semantic feature space and the segmentation probability space, resulting in improved accuracy without increasing model complexity. Numerous experiments demonstrated that the proposed method achieves the best performance on the widely used NYUv2 and SUN-RGBD datasets. And the effectiveness of our method in different scenes is verified on the outdoor scene dataset CamVid.