Thermal image-guided complementary masking with multiscale fusion for multi-spectral image semantic segmentation

Semantic segmentation of visible and thermal image is a kind of significant method for harsh environments. Most existing works focus on designing a multi-modal feature fusion module. However, these works may result in over-dependence on a specific modality and a lack of consideration for local and global context-aware information. Motivated by these issues, (1) a thermal image-guided complementary masking strategy is proposed to encourage the network to focus on regions with abundant semantic information; (2) a multi-modal fusion module is developed to integrate both local and global information and ensure consistency for semantic segmentation; (3) a self-distillation loss between unmasked and masked input modalities is introduced to enhance the robustness and consistency of the network. Particularly, the proposed masking strategy can force the network to concentrate on the meaningful area in all modalities, and thus the network can enhance the ability to connect context information. Experimental results on three public datasets demonstrate the superiority of our model.