PromptMID: Modal invariant descriptors based on diffusion and vision foundation models for optical-SAR image matching

Abstract

The ideal goal of generalizable image matching is to achieve stable and efficient performance in unseen domains. However, many existing learning-based optical-SAR image matching methods, despite demonstrating effectiveness in specific scenarios, often exhibit limited generalization and face challenges in adapting to practical applications. Repeatedly training or fine-tuning matching models to address domain differences not only lacks elegance but also incurs additional computational overhead and data production costs. In recent years, foundation models have shown significant potential for enhancing generalization. However, the disparity in visual domains between natural and remote sensing images poses challenges for their direct application. Consequently, effectively leveraging foundation models to improve the generalization of optical-SAR image matching remains a critical challenge. To address these challenges, we propose PromptMID, a novel approach that constructs modality invariant descriptors using text prompts based on land use classification as priors information for optical and SAR image matching. PromptMID consists of several key stages. Firstly, we fine-tune the diffusion model (DM) using we collected optical images, SAR images, and text prompts data to obtain the PromptDM model. Secondly, we construct modality-invariant descriptors by integrating multi-scale latent diffusion features extracted from the fine-tuned PromptDM model with multi-scale features derived from pre-trained visual foundation models (VFMs). To efficiently fuse local–global and texture-semantic features of varying granularities, we design a feature aggregation module (FAM) that ensures comprehensive feature representation. Finally, the discriminative power of the descriptors is enhanced through contrastive learning loss functions, aiming to improve the robustness and generalization of matching. Extensive experiments conducted on optical-SAR image datasets from five diverse regions demonstrate that PromptMID outperforms state-of-the-art matching methods, achieving superior performance in both seen and unseen domains while exhibiting strong cross-domain generalization capabilities. The source code will be made publicly available https://github.com/HanNieWHU/PromptMID.