Attention-based three-branch network for RGB-D indoor semantic segmentation

In indoor scene segmentation, utilizing the complementary information from RGB and depth images has demonstrated robustness and effectiveness in semantic segmentation. However, simple methods such as concatenating RGB and depth features or performing element-wise addition do not fully leverage the potential of multi-modal features. To better integrate these features, an attention-based three-branch RGB-D semantic segmentation network for indoor scenes, named ABTNet is proposed in this paper. First, this network employs a three-branch encoder architecture to extract RGB features, depth features, and fused features, effectively capturing important information while retaining the original RGB-D characteristics. Second, two modules include the Multi-modal Feature Fusion Module (MFFM) and the Multi-level Feature Refinement Module (MFRM) are presented. The MFFM filters RGB and depth features and performs adaptive fusion, while the MFRM achieves high-resolution predictions by integrating features from different levels. Experimental results demonstrate that the proposed model achieves excellent performance on both the NYUDv2 dataset and the more complex SUN-RGBD dataset.