Robust Skeletal-Graph Reconstruction Using mmWave Radar and its Application for Human-Activity Recognition

Abstract

Skeletal graphs can represent concise and reliable features for human-activity recognition in recent years. However, they have to be acquired by Kinect sensors or regular cameras, which rely on sufficient lighting. Meanwhile, skeletal graphs can only be created from the front views of sensors and cameras in the absence of any obstacle. The above stated restrictions limit the practical applicability of skeletal graphs. Therefore, in this work, we would like to investigate robust skeletal-graph reconstruction using milimeter-wave (mmWave) radar. The mmWave radar, which does not require light-of-sight propagation for data acquisition, can be equipped anywhere in room and operates in darkness so that it can overcome the aforementioned drawbacks. In this work, we propose to utilize the double-view cumulative numbers of radar-cloud points, temporal differentials in cumulative numbers of radar-cloud points, and Doppler velocities as the input features and adopt the deep-learning network integrating convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM). To fully investigate the effectiveness of our proposed new deep-learning network for robust skeletal-graph reconstruction, we evaluate the reconstruction accuracies in terms of mean absolute errorssubject to the human-location and human-orientation mismatches between the training and testing stages. Furthermore, we also investigate the advantage of our proposed novel robust skeletal-graph reconstruction approach in human-activity recognition since human-activity recognition turns out to be a primary application of skeletal graphs. We also compare the performances of our proposed new approach and two prevalent methods, namely, mmPose-natural language processing and BiLSTM in conjunction with CNN using the 3-D coordinates, signal-to-noise ratios, and Doppler velocites as the input features. Our experiments show that our proposed new approach outperforms the aforementioned two existing methods in both skeletal-graph reconstruction and human-activity recognition.