SDATFuse: Sparse dual aggregation transformer-based network for infrared and visible image fusion

Abstract

Infrared and visible image fusion aims to integrate complementary thermal radiation and detailed information to enhance scene understanding. Transformer architectures have shown promising performance in this field, but their feed-forward networks struggle to model multi-scale features, and self-attention often aggregates features using the similarities of all tokens in the queries and keys, which leads to irrelevant tokens introducing noise. To address these issues, this paper proposes a Sparse Dual Aggregation Transformer-based network for Infrared and Visible Image Fusion (SDATFuse). First, a hybrid multi-scale feed-forward network (HMSF) is introduced to effectively model multi-scale information and extract cross-modal features. Next, a sparse spatial self-attention mechanism is developed, using dynamic top-k selection operator to filter key self-attention values. By applying sparse spatial self-attention and channel self-attention in consecutive Transformer blocks, SDATFuse constructs a dual aggregation structure that efficiently integrates inter-block features. Additionally, a Dynamic Interaction Module (DIM) aggregates intra-block features across different self-attention dimensions. Finally, in the fusion stage, a Dual Selective Attention Module (DSAM) dynamically selects weights for global and local features from both modalities, utilizing spatial and channel self-attention maps. The proposed SDATFuse demonstrates superior performance on multiple infrared and visible image datasets. Experiments show that SDATFuse's fused results outperform state-of-the-art models in both qualitative and quantitative evaluations, effectively reducing noise and preserving detailed information.