Deep unfolding for hyper sharpening using a high-frequency injection module

Abstract

The fusion of multi-source data with different spatial and spectral resolutions is a crucial task in many remote sensing and computer vision applications. Model-based fusion methods are more interpretable and. flexible than pure data-driven networks, but their performance depends greatly on the established fusion model and. the hand-crafted, prior. In this work, we propose an end-to-end trainable model-based. network for hyperspectral and panchromatic image fusion. We introduce an energy functional that takes into account classical observation models and. incorporates a high-frequency injection constraint. The resulting optimization function is solved by a forward-backward splitting algorithm and. unfolded into a deep-learning framework that uses two modules trained, in parallel to ensure both data observation fitting and constraint compliance. Extensive experiments are conducted, on the remote-sensing hyperspectral PRISMA dataset and on the CAVE dataset, proving the superiority of the proposed deep unfolding network qualitatively and quantitatively.