Cross-modal 2D-3D feature matching: simultaneous local feature description and detection across images and point clouds

Abstract

Establishing correspondences between 2D images and 3D models is essential for precise 3D modeling and accurate positioning. However, widely adopted techniques for aligning 2D images with 3D features heavily depend on dense 3D reconstructions, which not only incur significant computational demands but also tend to exhibit reduced accuracy in texture-poor environments. In this study, we propose a novel method that combines local feature description and detection to enable direct and automatic alignment of 2D images with 3D models. Our approach utilizes a twin convolutional network architecture to process images and 3D data, generating respective feature maps. To address the non-uniform distribution of pixel and spatial point densities, we introduce an ultra-wide perception mechanism to expand the receptive field of image convolution kernels. Next, we apply a non-local maximum suppression criterion to concurrently evaluate the salience of pixels and 3D points. Additionally, we design an adaptive weight optimization loss function that dynamically guides learning objectives toward sample similarity. We rigorously validate our approach on multiple datasets, and our findings demonstrate successful co-extraction of cross-modal feature points. Through comprehensive 2D-3D feature matching experiments, we benchmark our method against several state-of-the-art techniques from recent literature. The results show that our method outperforms nearly all evaluated metrics, underscoring its effectiveness.