Fusion-Decomposition Pan-Sharpening Network With Interactive Learning of Representation Graph

Abstract

Deep learning (DL)-based pan-sharpening methods have become mainstream due to their exceptional performance. However, the lack of ground truth for supervised learning forces most DL-based methods to use pseudoground truth multispectral images, limiting learning potential and constraining the model's solution space. Unsupervised methods often neglect mutual learning across modalities, leading to insufficient spatial details in pan-sharpened images. To address these issues, this study proposes a fusion-decomposition pan-sharpening model based on interactive learning of representation graphs. This model considers both the compression process from source images to fused results and the decomposition process back to the source images. It aims to leveraging feature consistency between these processes to enhance the spatial and spectral consistency learned by the fusion network in a data-driven manner. Specifically, the fusion network incorporates the meticulously designed representational graph interaction module and the graph interaction fusion module. These modules construct a representational graph structure for cross-modal feature communication, generating a global representation that guides the cross-modal semantic aggregation of multispectral and panchromatic data. In the decomposition network, the spatial structure perception module and the spectral feature extraction module, designed based on the attributes of the source image features, enable the network to better perceive and reconstruct multispectral and panchromatic data from the fused result. This, in turn, enhances the fusion network's perception of spectral information and spatial structure. Qualitative and quantitative results on the IKONOS, GaoFen-2, WorldView-2, and WorldView-3 datasets validate the effectiveness of the proposed method in comparison to other state-of-the-art methods.