Enhancing mixture-of-experts model with prior knowledge for infrared and visible image fusion in complex degraded environments

Abstract

Infrared and visible image fusion aims to generate a composite image that simultaneously preserves thermal radiation information from infrared images and the rich texture details of visible images. However, existing studies have overlooked the adverse effects of scene degradation in visible images on the fusion process, leading to suboptimal fusion outcomes. To address the challenges posed by scene degradation in image fusion tasks, this paper proposes an image fusion network with degradation correction capability named the Enhancing Mixture-of-Experts model with Prior knowledge for infrared and visible image fusion (EMPFusion), which pioneers the automated execution of multiple degradation restoration tasks during the fusion process. First, we develop a diffusion model for degradation removal to generate high-quality pseudo-labels of visible images, thereby providing supervisory signals for training the fusion network. Second, to overcome the significant challenges in feature extraction caused by complex and diverse degradation scenarios, we design a Degradation removal backbone based on Prior knowledge and the Mixture-of-Experts (DPM) module. This architecture removes degradation with low loss and moderate computational overhead by integrating domain-specific prior knowledge and the Mixture-of-Experts framework. Furthermore, to mitigate semantic loss under extreme environmental conditions, we propose a Semantic Deconstruction and Segmentation (SDS) module based on image-text foundation models, enhancing semantic consistency throughout the fusion process. Extensive experiments demonstrate that EMPFusion excels in infrared-visible fusion tasks within complex degraded scenes. Across the LLVIP, M3FD, RoadScene, and MSRS datasets, EMPFusion achieves state-of-the-art (SOTA) performance on multiple evaluation metrics, showcasing exceptional degradation robustness and visual-semantic information preservation capabilities. By unifying adaptive degradation correction with fusion, this research addresses fusion distortion caused by degraded multimodal data in harsh environments, significantly enhancing applicability and robustness in downstream tasks such as autonomous driving and security monitoring.