Cross-attention interaction learning network for multi-model image fusion via transformer

Abstract

Current image fusion techniques often fail to adequately consider the inherent correlations among different modalities, resulting in suboptimal integration of multi-modal information. Drawing inspiration from inter-modal interactions, this paper introduces a cross-attention interaction learning network, CrossATF, leveraging the transformer architecture. The cornerstone of CrossATF resides in a generator network equipped with dual encoders. The multi-modal encoder incorporates two transformer modules of comparable computational complexity, alongside a meticulously designed cross-modal transformer. This architectural choice empowers the model to effectively extract modality-specific features while simultaneously integrating complementary features from diverse modalities. Furthermore, an auxiliary encoder is enlisted to encode features from the entire input image, thereby enhancing the model's comprehensive understanding of the image. Significantly, the loss function is tailored to selectively preserve a more targeted set of information from the source images, endowing the network with heightened feature extraction capabilities. Comprehensive experimental results across various datasets substantiate the promising performance of the proposed approach when compared to both task-specific methodologies and unified fusion frameworks.