SfM on-the-fly: A robust near real-time SfM for spatiotemporally disordered high-resolution imagery from multiple agents

Abstract

In the last twenty years, Structure from Motion (SfM) has been a constant research hotspot in the fields of photogrammetry, computer vision, robotics etc., whereas real-time performance has only recently emerged as a topic of growing interest. This work builds upon the original on-the-fly SfM (Zhan et al., 2024) and presents an updated version (v2) with three new advancements to get better SfM reconstruction results during image capturing: (i) near real-time image matching is further boosted by employing the Hierarchical Navigable Small World (HNSW) graphs, and more true positive overlapping image candidates can be faster identified; (ii) a self-adaptive weighting strategy is proposed for robust hierarchical local bundle adjustment to improve the SfM results; (iii) multiple agents are included for supporting collaborative SfM and seamlessly merge multiple 3D reconstructions into a complete 3D scene in presence of commonly registered images. Various comprehensive experiments demonstrate that the proposed SfM method (named on-the-fly SfMv2) can generate more complete and robust 3D reconstructions in a time-efficient way. Code is available at http://yifeiyu225.github.io/on-the-flySfMv2.github.io/.