Multi-scale subtraction and attention-guided network for RGB-D indoor scene parsing

RGB-D scene parsing is a fundamental task in computer vision. However, the lower quality of depth images often leads to less accurate feature representations in the depth branch. Additionally, existing multi-level fusion methods in decoders typically use a unified module to merge RGB and depth features, disregarding the unique characteristics of hierarchical features. This indiscriminate approach can degrade segmentation accuracy. Hence, we propose a Multi-Scale Subtraction and Attention-Guided Network (MSANet). Firstly, through a cross-modal fusion module, we fuse RGB and depth features along the horizontal and vertical directions to capture positional information between the two modalities. Then, we use a Spatial Fusion Unit to adaptively enhance depth and RGB features spatially. Furthermore, we analyze the feature differences across various decoder levels and divide them into spatial and semantic branches. In the semantic branch, a high-level cross-modal fusion module extracts deep semantic information from adjacent high-level features through backpropagation, enabling RGB and depth layer reconstruction and mitigating information disparity and hierarchical differences with subtraction operations. In the spatial branch, a low-level cross-modal fusion module leverages spatial attention to enhance regional accuracy and reduce noise. MSANet achieves 52.0% mIoU on the NYU Depth v2 dataset, outperforming the baseline by 5.1%. On the more challenging SUN RGB-D dataset, MSANet achieves 49.0% mIoU. On the ScanNetV2 dataset, MSANet achieves 60.0% mIoU, further validating its effectiveness in complex indoor scenes.