Cross-Modal Collaborative Evolution Reinforced by Semantic Coupling for Image Registration and Fusion

Abstract

Joint image registration and fusion aim to align and integrate source images to generate an image with salient targets and rich texture details. Current methods pursue spatially optimal deformation fields. However, these methods often overlook local semantic alignment, leading to exacerbated heterogeneity in cascaded fusion and vision tasks. To address these issues, we propose a collaborative evolution network reinforced by semantic coupling for image registration and fusion, named CE-SCNet. First, to correct spatial misalignments, we design a multiscale deformation estimator (MSDE). This module is to estimate spatial deformation fields by modeling global relationships across multiple scales. Second, to further enhance semantic alignment and mitigate heterogeneity, we design the semantic interaction module (SIM). This module is to integrate contextual information within the semantic domain for feature coupling. Third, to reconstruct images with high visual perception, we design the feature discrimination module (FDM) and the detail awareness module (DAM). Both modules are to capture texture information from multiple perspectives. Finally, to optimize the joint paradigm, we construct a multilabel semantic loss. Extensive experimental validations have shown that CE-SCNet significantly outperforms state-of-the-art methods in alleviating semantic misalignments. The semantic segmentation experiments demonstrate that CE-SCNet can adapt to the semantic demands of high-level vision tasks.