SAM-guided multi-level collaborative Transformer for infrared and visible image fusion

Abstract

The primary value of image fusion lies in supporting downstream task more effectively. However, the fusion representation of existing methods contains insufficient semantic information, thereby weakening the compatibility with subsequent task. To overcome this problem, a SAM-guided multi-level collaborative transformer for infrared and visible image fusion is proposed in this manuscript, termed as SpTFuse. Considering the strong zero-shot generalization ability of Segment Anything Model (SAM), a SAM-based semantic prior branch is introduced to interact with multi-scale visual representation branches for improving the completeness and compatibility of fusion representation. The interaction process is divided into three levels to progressively integrate multibranch information. At the first level, an inter-modal fusion block (IEB) is designed with a single-step collaborative transformer (SCT) and a modality integration module (MIM). The SCT aggregates the correlated features of semantic prior and visual representation. Then, the MIM is designed to fuse the SAM semantic prior guided multimodal visual representation. To balance the visual and semantic representations to obtain complete fusion representation, an intra-modal interaction block (IAB) is constructed at the following levels. Specifically, the IAB consists of a dual-path collaborative transformer (DCT) and a semantic enhancement module (SEM). The DCT constructs two paths in a cascade manner, where the prior collaborative path continues to acquire semantic prior, while the visual refinement path balances visual information while maintaining semantic completeness. Subsequently, SEM further combines semantic prior to enhance the completeness of the fused representation. To reduce the semantic information discarded during the image restoration process, the collaborative information of previous levels is incorporated into the corresponding decoder layers by the semantic compensation block. Finally, the proposed loss function includes semantic prior loss, gradient loss, and intensity loss. The experiments demonstrate the SpTFuse not only achieves effective fusion results, but also shows obvious advantages in downstream tasks such as segmentation and detection. The source code is available at https://github.com/lxq-jnu/SpTFuse.