Fusion of LiDAR and Camera Images in End-to-end Deep Learning for Steering an Off-road Unmanned Ground Vehicle

We consider the task of learning the steering policy based on deep learning for an off-road autonomous vehicle. The goal is to train a system in an end-to-end fashion to make steering predictions from input images delivered by a single optical camera and a LiDAR sensor. To achieve this, we propose a neural network-based information fusion approach and study several architectures. In one study focusing on late fusion, we investigate a system comprising two convolutional networks and a fully-connected network. The convolutional nets are trained on camera images and LiDAR images, respectively, whereas the fully-connected net is trained on combined features from each of these networks. Our experimental results show that fusing image and LiDAR information yields more accurate steering predictions on our dataset, than considering each data source separately. In another study we consider several architectures performing early fusion that circumvent the expensive full concatenation at raw image level. Even though the proposed early fusion approaches performed better than unimodal systems, they were significantly inferior to the best system based on late fusion. Overall, through fusion of camera and LiDAR images in an off-road setting, the normalized RMSE errors can be brought down to a region comparable to that for on-road environments.