Toward Aerial Collaborative Stereo: Real-Time Cross-Camera Feature Association and Relative Pose Estimation for UAVs

The collaborative unmanned aerial vehicles (UAVs) can construct a wide and variable baseline stereo camera, providing potential benefits in flexible and large-scale depth perception. Compared to the fixed-baseline stereo camera, the collaborative stereo system in dynamic flight face the additional challenge of continuously varying baselines, which necessitates real-time cross-camera stereo feature association and real-time relative pose estimation on resource-constrained onboard computers. To tackle these challenges, we propose a real-time dual-channel feature association with a guidance-prediction framework. This framework utilizes a graph neural network (GNN) to periodically guide cross-camera feature associations between two UAVs, while the guided features are continuously predicted to enable real-time feature association. Additionally, we propose a real-time relative multistate-constrained Kalman filter (Rel-MSCKF) algorithm, which efficiently integrates covisual overlapping features with the UAVs’ visual-inertial odometry (VIO), enabling accurate and fast relative pose estimation. Extensive real-world experiments are performed using the widely adopted but resource-constrained NVIDIA NX onboard computer. The results demonstrate that the dual-channel algorithm achieves cross-camera feature association for 30 Hz image streams, with an average runtime of 14 ms, significantly outperforming the conventional cross-camera feature matching algorithms, which typically require over 70 ms. For the relative pose estimation of two cameras, the proposed Rel-MSCKF algorithm achieves pose estimation within 16 ms, outperforming the current pose-graph optimization (PGO) manner with 330 ms. Additionally, we examine the convergence behavior of Rel-MSCKF under various spatial configurations. The system’s robustness is further evaluated under the challenges of asynchronous image acquisition, communication interruptions, and field-of-view (FOV) occlusions. Online video: https://youtu.be/avxMuOf5Qcw