Efficient Human Action Recognition With Fine-Grained Spatiotemporal Feature Extraction From Millimeter-Wave Point Clouds

Abstract

Human activity recognition (HAR) based on millimeter-wave (mmWave) radar point clouds has attracted much attention due to its privacy protection properties. The point cloud sequence generated by mmWave radar contains the appearance and motion features of objects and contains rich spatiotemporal information. However, due to the sparsity, nonuniformity, and noise interference of mmWave point clouds, existing methods had difficulty in effectively extracting fine-grained spatiotemporal features from point cloud sequences. To address these problems, we propose a new HAR system for mmWave radar point clouds that can effectively extract fine-grained spatiotemporal features in point cloud sequences and significantly reduce computational overhead. Our system first preprocesses the raw point cloud to generate a clean and standardized point cloud. Then, it uses shared weight TF-Net and PointNet++ to extract features and centroid coordinates for each point cloud frame and inputs them into our designed ST-Transformer layer. This layer decouples and encodes the spatiotemporal structure of the centroid coordinates to capture fine-grained spatiotemporal information. Finally, a lightweight neural network based on a multilayer perceptron (MLP) performs classification. The whole process avoids voxelization, reducing memory requirements and computational complexity. We conduct extensive experiments on RadHAR and Pantomime datasets to evaluate the effectiveness of the proposed system, achieving average recognition accuracies of 98.8% and 99.1%, respectively, which is detailed in the Experimental Results section.