SamFusion:A model for multimodal image fusion guided by SAM’s rich semantics

Abstract

Infrared and visible image fusion aims to combine the advantages of each modality, such as salient targets in infrared images and detailed textures in visible images, in order to generate images with better visual effects and richer information. Existing fusion methods do not distinguish between semantic and texture information, ignoring their specific roles in the image fusion process, whereas we have demonstrated the critical role of semantic information in the image fusion task. However, the semantic information extraction networks of current methods are restricted to be trained on limited infrared and visible datasets, which results in significantly inadequate semantic information extraction capabilities when facing complex scenes. To address this challenge, we pioneered SamFusion, an image fusion model that leverages the SAM large model, which effectively exploits the rich semantic information in SAM to facilitate image fusion. Specifically, we introduce the Multi Level Semantic Aggregation (MLSA) and Soft Interaction (SI) modules, which aim to mine the semantic information at each level in SAM that is conducive to fusion, while reducing the modality differences in semantic information. In addition, specialized fusion modules are designed for semantic information and texture information respectively. For texture information with significant modality differences, semantic information is used as a guide for selective fusion. Extensive experiments show that our method outperforms the state-of-the-art methods in highlighting salient targets and detailed textures. In addition, the effectiveness of our method is revealed in medical image fusion and downstream tasks of object detection and semantic segmentation.