Novel vision-LiDAR fusion framework for human action recognition based on dynamic lateral connection

In the past decades, substantial progress has been made in human action recognition. However, most existing studies and datasets for human action recognition utilise still images or videos as the primary modality. Image-based approaches can be easily impacted by adverse environmental conditions. In this paper, the authors propose combining RGB images and point clouds from LiDAR sensors for human action recognition. A dynamic lateral convolutional network (DLCN) is proposed to fuse features from multi-modalities. The RGB features and the geometric information from the point clouds closely interact with each other in the DLCN, which is complementary in action recognition. The experimental results on the JRDB-Act dataset demonstrate that the proposed DLCN outperforms the state-of-the-art approaches of human action recognition. The authors show the potential of the proposed DLCN in various complex scenarios, which is highly valuable in real-world applications.