DGTF: A framework utilizing textual semantics for detail-aware and global fusion with a multi-degradation scenarios dataset

Abstract

Infrared and visible light fusion aims to integrate information from both modalities to generate high-quality fused images. However, existing methods do not perform well in multiple degraded scenarios. In these scenarios, the source images face quality degradation and information loss. Fusion combined with degradation processing often damages the detailed target information in the images. To address this limitation, this paper proposes a novel Degradation-Text Fusion framework, named DGTF, which leverages cascaded degradation text, object text, and target masks for detail-aware degradation regulation. The framework adjusts the details based on the object text specified in the given input, ensuring that global degradation processing does not compromise the quality of detail fusion. This approach overcomes the limitations of previous methods constrained by global degradation processing. To train and evaluate DGTF, we constructed a new infrared and visible light dataset, Multi-degraded scene text target infrared and visible datasets (MTS), which encompasses seven extreme scenarios, including rain, snow, fog, low light, exposure, infrared noise, and low contrast. Extensive experimental results demonstrate that our method significantly outperforms existing techniques in fusion performance, even without text guidance. Furthermore, tests conducted on the MTS dataset reveal that the detail-regulated fusion results achieved by DGTF far surpass traditional degradation-based fusion methods, effectively enhancing the performance of advanced vision tasks. These findings validate the effectiveness of the proposed detail regulation framework. Our code is available at https://github.com/linshenj/DGTF.