Segment anything model edge prior-guided infrared and visible image fusion method

Abstract

Hindered by the low robustness of visible imaging and the low resolution of infrared imaging, visible and infrared images captured in real scenarios often suffer from severe detail loss and indistinct edge appearance. Therefore, modeling of fine-grained edges is crucial for high-quality fused image generation. However, recent deep learning-based fusion algorithms have limitations in exploring highly robust structural priors, in other words, limited edge perception ability, resulting in edge artifacts and low clarity. To address the aforementioned issues, we propose a novel fusion framework guided by segment anything model (SAM) edge priors. SAM boasts outstanding zero-shot generalization capabilities, enabling it to extract high-quality edge priors from target scenes even under unideal imaging conditions, such as low-light and dense noise. We propose an image content-edge fusion block (CEFB), which progressively injects edge information from the source images into the image content features, to enhance their edge representation ability. Additionally, considering the significance of inter-modality interaction, we also introduced an image translation network, achieving mutual translation between infrared and visible modalities. Then, the SAM edge priors extracted from the transformed images, which have unchanged content but altered style, are utilized and embedded into the content features after modality interaction through the proposed CEFB, thus strengthening the representation of modality-invariant edge structures. Extensive experiments on two public datasets demonstrate our method generates fused images with more distinct edges and enhanced target information, and also exhibits strong generalization capabilities across diverse scenarios.