Modal-invariant progressive representation for multimodal image registration

Many applications, such as autonomous driving, rely heavily on multimodal data. However, differences in resolution, viewing angle, and optical path structure cause pixel misalignment between multimodal images, leading to distortions in the fusion result and edge artifacts. In addition to the widely used manual calibration, learning-based methods typically employ a two-stage registration process, referred to as “translating-then-registering”. However, the gap between modalities makes this approach less cohesive. It introduces more uncertainty during registration, misleading feature alignment at different locations and limiting the accuracy of the deformation field. To tackle these challenges, we introduce the Modality-Invariant Progressive Representation (MIPR) approach. The key behind MIPR is to decouple features from different modalities into a modality-invariant domain based on frequency bands, followed by a progressive correction at multiple feature scales. Specifically, MIPR consists two main components: the Field Adaptive Fusion (FAF) module and the Progressive Field Estimation (PFE) module. FAF integrates all previous multi-scale deformation subfields. PFE progressively estimates the remaining deformation subfields at different scales. Furthermore, we propose a two-stage pretraining strategy for end-to-end registration. Our approach is simple and robust, achieving impressive visual results in several benchmark tasks, even surpassing the ground truth from manual calibration, and advancing downstream tasks.