Infrared and visible image fusion based on text-image core-semantic alignment and interaction

Abstract

The text prior can effectively compensate for the limitations of image modality in capturing semantic information, which makes the fusion process more semantic and contextual. However, current fusion methods are not sufficiently adaptive to flexible text inputs and lack the precise alignment between textual semantics and image local regions. To address these issues, an image fusion method based on the text-image core-semantic alignment and interaction is proposed to bridge the gap between cross-modal information. The text-image core-semantic alignment module is designed to refine the close adherence between text and object regions through a pixel-wise coarse-to-fine segmentation mechanism. Meanwhile, a synergistic fusion pipeline is devised to establish a link between a contextual feature extraction unit and a cross-modal affine fusion module. The pipeline directs local attention to text-adherent image regions, while coupling global text features to compensate for the contextual details of whole images. In this way, the fused images enhance the adherence to the flexible text and capture richer contextual details for a more comprehensive visual representation. Extensive experiments on several datasets demonstrate that the proposed text-guided fusion method has obvious advantages over state-of-the-art methods in fusion performance.