Exploiting Joint Representation via Mixture of Experts for Pansharpening

Abstract

Pansharpening, which aims to reconstruct high-resolution multispectral images (MSIs) by combining low-resolution multispectral observations with high-resolution panchromatic images (PAN), is crucial for overcoming satellite bandwidth limitations. However, existing pansharpening methods typically adopt one of two strategies: 1) employing dual-branch or multibranch architectures with separate feature extraction pipelines for each remote sensing image, which limits their ability to fully exploit synergistic potential; or 2) utilizing a single-branch or shared module approach, which hinders the capture of diverse, source-specific representations. To address these limitations, we propose a novel pansharpening method denoted JRMoE, which incorporates a mixture-of-experts (MoE) framework. JRMoE includes a router that dynamically selects a combination of experts for each input sample, enabling the model to capture both unique and complementary information from different sources. Furthermore, to ensure balanced expert utilization and preserve functional diversity, we introduce a complementarity enhancer and a uniqueness amplifier. The complementarity enhancer prevents overreliance on specific experts, while the uniqueness amplifier encourages individual experts to specialize in distinct aspects of the data, thereby promoting the preservation of distinctive characteristics in the learned representations. Extensive experiments on GF-2, QB, and WV-III benchmarks show that JRMoE consistently outperforms recent pansharpening methods, yielding average gains of 3.77%, 2.96%, and 0.54% in ERGAS, SAM, and Q2n, respectively, over the strongest competing methods on the three test sets. All implementations of this project are released at https://github.com/JUSTM0VE0N/RSMIF_Project