Fusing Monocular RGB Images With AIS Data to Create a 3-D Bounding Box Estimation Data Set for Marine Vessels

Abstract

This article presents a novel technique for creating a 3-D bounding box estimation data set for marine vessels by fusing monocular RGB images with automatic identification system (AIS) data. The proposed technique addresses the limitations of relying purely on AIS for location information, caused by issues like equipment reliability, data manipulation, and transmission delays. By combining vessel detections from monocular RGB images, obtained using an object detection network (YOLOX-X), with AIS messages, the technique generates 3-D bounding boxes that represent the vessels’ location, orientation, and dimensions. The article evaluates different object detection models to locate vessels in image space. We also compare two transformation methods [homography and perspective-n-point (PnP)] for aligning AIS data with image coordinates. The results of our work demonstrate that the PnP method achieves a significantly lower projection error compared to homography-based approaches used before, and the YOLOX-X model achieves a mean average precision of 0.80 at an Intersection over Union threshold of 0.5 for relevant vessel classes. We show that our approach allows the creation of a 3-D bounding box estimation data set without needing manual annotation. In addition, we introduce the Boats on Nordelbe Kehrwieder, a publicly available data set comprising 3753 images with 3-D bounding box annotations, created by our data fusion approach. This data set can be used primarily for benchmarking 3-D bounding box estimation networks. In addition, we introduce a set of 1000 images with 2-D bounding box annotations for ship detection from the same scene.