A multispectral pansharpening method based on CNN-DI network with mixture of experts

Abstract

The process of fusing two complementary data, panchromatic and multispectral images, to create high-resolution multispectral (HRMS) images is known as pansharpening. Combining detail injection (DI) methods with convolutional neural networks (CNN) for improved HRMS image fusion quality is a research hotspot due to their interpretability and large-scale data processing capabilities, respectively. Nevertheless, the current hybrid models typically concatenate CNN and traditional techniques, limiting the ability to utilize the benefits of both approaches. This paper presents a new hybrid network, multispectral pansharpening method based on CNN-DI network with mixture of experts (CDN-MoE), using detail injection theory to design a deep learning framework. Specifically, we first create the mixture of detail inject experts network (MoDIE-Net) that mixes training pairs of full- and reduced-resolution images to enhance model generalization. Next, the adaptive correlation residual network (ACR-Net) is suggested to find the correlation between the spectral and spatial features of the source images. Finally, the global information injection network (GII-Net) is established to strengthen the accuracy of fusion results by integrating the context of input images. Additionally, to reduce the loss of spectral features during the upsampling process, the spectral reconstruction network (SR-Net) is proposed. We perform both qualitative and quantitative experiments on the GaoFen-2, IKONOS, and WorldView-2 datasets at various resolutions. Our approach has advantages over other SOTA pansharpening methods currently available in terms of visual effects and objective metrics.