NTFNet: Narrowing-Then-Fusing network for RGB-TIR semantic segmentation

In recent years, the task of understanding scenes in visible (RGB) and thermal-infrared (TIR) images has garnered increasing interest in the field of computer vision. However, most existing methods employ simplistic fusion strategies to merge features from different modalities. These strategies often overlook the differences in shallow-level features between modalities, thereby reducing the discriminability of the fused features and resulting in suboptimal segmentation performance. To address this issue, we present a novel RGB-TIR semantic segmentation framework, named NTFNet. This framework exploits the potential consistency of semantic-level features to rectify shallow-level features and reduce discrepancies between modalities prior to integration. Specifically, auxiliary encoders are employed at each layer to capture semantically consistent information. To obtain rich multi-modal semantic features, we designed a High-Level Feature Fusion Module (HFFM) that enhances feature representation in both channel and spatial dimensions. Subsequently, the Shallow Feature Difference Rectification Module (SFDRM) is introduced to rectify the difference in shallow-level features. To address the loss of detailed information during the rectification process, the SFDRM incorporates a Detail Attention Mechanism (DAM) to preserve the original detail information, thereby further optimizing the final segmentation results. In the end, a Multi-Scale Feature Fusion module (Multi-Scale FFM) is designed to combine the rectified features. Comprehensive experiments on two public RGB-TIR datasets show that our method significantly outperforms other state-of-the-art approaches in terms of performance.