Total solution for simultaneous pose and correspondence estimation of drone images in urban environments

Abstract

Vision-based pose estimation for drone images in urban environments is particularly challenging when reliable GNSS and IMU signals are unavailable and the search space spans large areas. Traditional methods depend on known correspondences of well-defined landmark objects, which are not always feasible in complex urban environments. To address this problem, we propose a total solution that simultaneously estimates the image pose and its correspondences to a semantic map database. A cascaded network, named dual-head SegFormer, is developed to generate multi-class semantic segmentation maps and high-quality road centerlines from images. A city-wide coarse-to-fine image localization strategy aligns the image segmentation map with the database map using class-label consistency and graph representation indices, yielding initial poses for further optimization. The final pose is determined by minimizing a novel objective function that evaluates the differences between the image and database across three key aspects: semantic maps, road attributes, and tie point reprojection errors. Evaluated on three urban drone image datasets, our method achieves position and rotation errors below 2.03 m and 2 ${}^{°}$ relative to the bundle adjustment results. By incorporating semantic features and an improved objective function, our method achieves notable enhancements in robustness and accuracy compared to prior approach that relied exclusively on road attributes. This work provides a dependable alternative for vision-based navigation, further reducing dependence on GNSS data or precise initial pose information.