Multi-source perception data fusion of vessels in visual occlusion scenarios: Leveraging prior knowledge of vessel motion

Automatic Identification System (AIS) and cameras are widely used in harbor and coastal supervision to detect ship movement. Integrating both can enhance the perception and monitoring of the navigation status of surrounding vessels. However, their integration faces several challenges. First, the two data sources have different coordinate systems and sampling frequencies. Additionally, in video data, visual occlusion during vessel encounters may lead to the loss or displacement of ship detection targets, significantly affecting the fusion of ship data. To address these issues, we incorporate prior knowledge of vessel motion to strengthen the model’s ability to identify and track occluded targets. Firstly, when detecting and tracking ships in video, we use occlusion prior knowledge and tracking results to evaluate and manage the occluded detection box areas. Secondly, when the occluded detection box disappears or undergoes severe deformation, we use the prior knowledge of ship motion characteristics from AIS and images to predict the detection box. Finally, we validated our improved method on the FVessel_v1.0 dataset, confirming its accuracy in data fusion under occlusion conditions. Compared with the state-of-the-art ship data fusion algorithm, our method improved the Multiple Object Fusion Accuracy (MOFA), Identification Precision (IDP), Identification Recall (IDR), and Identification F1 score (IDPF1) metrics by 3.01%, 1.33%, 1.57%, and 1.33%, and reduced MOFA by 1.85%. Additionally, by only changing the method of utilizing prior knowledge without altering the detection and tracking algorithms of the state-of-the-art ship data fusion algorithm, we improved the MOFA, IDP, IDR, and IDPF1 metrics by 2.65%, 1.94%, 0.62%, and 0.87%, respectively, and reduced MOFA by 1.16%.