SwinEIF: Explicit and implicit Swin Transformer fusion for infrared and visible images

Abstract

Infrared and visible image fusion aims to integrate thermal target information from infrared images with fine texture details from visible images, enabling comprehensive scene perception in complex environments. Existing explicit methods rely on hand-crafted rules that struggle with cross-modal discrepancies, while implicit end-to-end models require large paired datasets that are often unavailable. To address these challenges, we propose SwinEIF, a Swin Transformer-based fusion framework that synergizes Explicit and Implicit paradigms for infrared-visible image Fusion. The framework innovatively combines an explicit unimodal feature extraction branch, which learns modality-specific representations through Swin Transformer’s hierarchical self-attention, with an implicit multi-modal feature interaction branch that facilitates adaptive feature fusion via cross-attention between modalities. Additionally, a Discrete Wavelet Transform (DWT)-based fusion decoder is incorporated to fuse features from different frequency bands, and this fusion process uses the content of the source images as fusion weights to generate end-to-end fusion outputs. Leveraging the strengths of the explicit image fusion paradigm, the unimodal feature extraction branch is trained on large-scale, unaligned infrared and visible images, enabling the network to capture diverse patterns and comprehensively extract global information in the first training stage. In the second stage, a smaller, aligned infrared-visible image dataset is then used to fine-tune the multi-modal feature interaction branch and the DWT-based fusion decoder, ensuring high-quality fusion outputs. By fully combining the advantages of both paradigms, SwinEIF demonstrates superior performance across multiple infrared-visible image datasets, outperforming state-of-the-art fusion methods. Experimental results confirm that SwinEIF excels in both subjective visual quality and objective evaluation metrics, showcasing remarkable fusion performance and strong generalization capabilities.