Registration-fusion binocular diffusion model: Exploring continuous fusion of unregistered hyperspectral and multispectral images

Abstract

The fusion of hyperspectral image (HSI) and multispectral image (MSI) constitutes an effective approach to obtain high spatial resolution HSI (HR-HSI). However, in practice, multi-source images obtained under different imaging conditions are difficult to be perfectly registered. The vast majority of existing fusion methods crudely assume that HSI and MSI are registered, which is challenging to generalize to practical applications. To this end, we propose a unified registration-fusion binocular diffusion model (RF-BDiff) to achieve continuous fusion of unregistered HSI and MSI. RF-BDiff alternately optimizes image registration and fusion tasks to gradually recover HR-HSI by repeated refinement over multi time steps. The refinement operation at each time step is performed by the specially designed implicit neural registration module (INRM) and invertible spatial–spectral fusion module (IS²FM). INRM parameterizes the mapping between spatial coordinates of MSI and spectral values of reconstructed image and HSI as continuous implicit functions to provide the registered spatial coordinates for subsequent refinement. IS²FM designs an invertible bijective transformation for modeling fusion problem of registered images to reduce the information loss in fusion process and enriches the spatial–spectral information of reconstructed image. INRM and IS²FM are executed alternately to continuous fuse unregistered HSI and MSI. Such a design is conducive to the generalization of RF-BDiff in real applications, and we especially provide a real dataset to verify its effectiveness in real scenarios. Systematic experiments on simulated and real datasets demonstrate the state-of-the-art performance of RF-BDiff.