Learning to navigate on the rough terrain: A multi-modal deep reinforcement learning approach

How to enable safe navigation of unmanned vehicles on complex and rough terrain is challenging and meaningful research. In this paper, we propose an end-to-end reinforcement learning local navigation method with multi-modal data fusion, which effectively combines the intrinsic perception, such as Inertial Measurement Unit (IMU) measurements, and the extrinsic perception, such as Three-dimensional (3D) point clouds and images, of an unmanned vehicle. A specific feature extraction network is constructed for each modal data, and the total network is effectively trained using a modal separation learning method. The experimental results show that the proposed method can effectively address various obstacles such as rough roads, vegetation obstacles, and water pool disturbances to achieve autonomous and safe navigation of unmanned vehicles in off-road scenarios.