One-stream one-stage pure transformer for infrared and visible image fusion

Abstract

Existing infrared and visible image fusion methods usually use two same backbones to extract deep features of the source images, and then manually design a fusion strategy to fuse them. However, this framework overlooks the importance of feature interaction during the feature extraction stage and does not fully capture the complementary information between the source images. In addition, it requires the design of complex fusion strategies, which limits its robustness and generalization to different scenarios. To this end, we propose a one-stream one-stage pure Transformer-based fusion framework, which simplifies feature extraction and fusion into a unified one-stream pipeline. Specifically, the proposed method consists of a one-stream fusion network and a decomposition network. The fusion network uses several Swin Transformer blocks to extract and fuse two modality features simultaneously. Thanks to the sliding-window-based multi-head attention, the fusion network can acquire local features and global dependencies and seamlessly model their contextual relationships. Due to the lack of effective supervision signals, the fusion network struggles to fully transfer important information from the source images, which can easily lead to the generation of artifacts. To eliminate these artifacts and simultaneously force the fused image to contain richer information, we design a simple decomposition network that decomposes the fusion result into the source images with consistency constraints. Extensive comparative and ablation experiments on four image fusion benchmarks demonstrate that our method achieves favorable results. In addition, the results on downstream tasks, including object detection and semantic segmentation, further show the effectiveness of the proposed method.