Fast All-day 3D Object Detection Based on Multi-sensor Fusion

Abstract

Realtime 3D object detection in all-day conditions is a challenging task for autonomous vehicles. Various image and point cloud based object detection methods have been proposed. Image based detectors are sensitive to illumination changes and cannot get accurate 3D information. Point cloud based detectors are less convenient for acceleration and deployment on commonly used hardware due to the unstructured nature of point cloud data, making it prohibitive for mobile platforms with limited computing resources in large-scale outdoor scenes. Frustum based 3D object detector first detects 2D objects in the image and then extracts frustum point cloud according to the cross-calibration parameters. Small-scale frustum point clouds can be used for 3D object detection, hence substantially accelerating the computation. However, when objects are missed in the first stage image based detector, the whole algorithm will fail to detect them. In this paper, we extended the frustum based 3D object detector by leveraging more sensor modalities. Our method combines two frustum based 3D object detecting branches in which visible light image and thermal image are used for 2D ROI extracting respectively. After obtaining 3D object proposals from the two branches, 3D non-maximum suppression is conducted to get the final detections. Experiments tested on our experimental autonomous vehicle show that our proposed method is capable of detecting 3D objects fast in various complex environments.