Multimodal Remote Sensing Image Registration via Modality Perception and Self-Supervised Position Estimation

Abstract

Multimodal remote sensing image registration ensures that images from different sensors or modalities are spatial and informational consistent for effective comparison and analysis. However, due to the nonlinear modality gaps that exist between images, it is difficult to focus solely on spatial positional differences while ignoring the modality gaps. In this article, to address this issue, we propose a new framework for multimodal registration network, named MMRNet. The proposed framework comprises the following main aspects. First, a novel self-supervised positional misalignment estimator (PME) is designed for multimodal image registration. PME can efficiently overcome the modality gaps and learn the positional differences between multimodal images more reliably, optimizing the registration loss by minimizing the positional differences directly. Then, a new paradigm of modality translation, termed modality perception module (MPM), is introduced to effectively learn modality gaps and perform modality translation in the case of positional misalignment. Finally, we further design the modality perception guidance loss to supervise the modality translation task, which can encourage the fidelity of the generated pseudo-modality images. Our registration network integrates both rigid registration model and nonrigid registration model. The experimental results demonstrate that the proposed registration framework can obtain obviously superior performance in both rigid and nonrigid image registration tasks on optical-synthetic aperture radar (SAR) data, optical-map data, and optical-infrared data. The code and relevant dataset will be made publicly available at https://github.com/Ahuer-Lei/MMRNet.