CDM-Net: A Framework for Cross-View Geo-Localization With Multimodal Data

Abstract

Cross-view geo-localization (CVGL) task aims to match images of the same object captured from various platforms, such as drones and satellites. The primary challenge of CVGL is that the flight altitude and shooting angle of drones can lead to changes in the visual appearance of the same target building. Most existing methods regard CVGL task as an image retrieval problem based on classification networks. Nevertheless, the multimodal information correspondence of drone-satellite views and the structural features of target buildings are not fully explored. In this article, we propose a novel CVGL framework based on multimodal information, named conditional diffusion matching network (CDM-Net). The framework consists of three stages: preprocessing, drone-view image synthesis, and structural feature matching. Specifically, the first stage employs perspective transformation (PT) to convert the drone’s oblique view into a vertical view and describes the remote sensing (RS) information of the drone images. In the second stage, we use a conditional diffusion (CD) model to synthesize a high-precision vertical view by combining the converted drone-view image with RS textual information as input. In the final stage, we utilize a structural feature matching network to extract robust point-line features of the target building, subsequently outputting the retrieval list. Experiments on two widely used public benchmarks, University-1652 and SUES-200, showed that the proposed method achieved competitive results. Additionally, our method also outperformed existing methods in cross-modal dataset generalization. The datasets of this study are available at https://github.com/cver6/CDM-Net