Radar-Camera Fusion for Ground-based Perception of Small UAV in Urban Air Mobility

The resilient surveillance of cooperative and non-cooperative aerial targets is critical for the safety and security of urban air mobility operations. Accurate detection, tracking, and trajectory prediction are essential to the subsequent tasks, e.g. tactical conflict prediction and resolution. Meanwhile, the combination of radar and camera is a classic option to provide perception services in different challenging environments. In this paper, a deep semantic association network is proposed for building relationships between the image detections and raw radar points, which then contributes to subsequent tasks, e.g. detecting, tracking, and predicting the small UAV with networked radar and camera systems. Various flight trials are conducted for collecting multi-sensor data, finally, training and testing results on this dataset demonstrate the outstanding performance of the proposed fusion workflow in comparison to single-sensor performance. At the same time, the 2D predictions in the sensor network are reconstructed to 3D trajectories for comparison and also reveal the improvements of the radar-camera fusion approach.