mmPrivPose3D: A RaDAR-Based Approach to Privacy-Compliant Pose Estimation and Gesture Command Recognition in Human–Robot Collaboration

Abstract

Various sensors are employed in dynamic human-robot collaboration manufacturing environments for real-time human pose estimation to improve safety through collision-avoidance systems and gesture command recognition to enhance human-robot interaction. However, the most widely used sensors—RGBD cameras—often underperform under varying lighting and environmental conditions and raise privacy concerns. This article introduces mmPrivPose3D, a novel system designed to prioritize privacy while performing human pose estimation and gesture command recognition using a 60-GHz industrial frequency-modulated continuous wave (FMCW) RaDAR with a 10-m maximum range and 29degrees angular resolution. The system employs a parallel architecture including a 3-D convolutional neural network (CNN) for pose estimation, which extracts 19 keypoints of the human skeleton, along with a random forest classifier for recognizing gesture commands. The system was trained on a dataset involving ten individuals performing various movements in a human-robot interaction context, including walking in the workspace and hand-waving gestures. Our model demonstrated a low-mean per joint position error (MPJPE) of 4.8% across keypoints for pose estimation and, for gesture recognition, an accuracy of 96.3% during $\mathsf {k}$ -fold cross validation and 96.2% during inference. mmPrivPose3D has the potential for application in human workspace localization and human-to-robot communication, particularly in contexts, where privacy is a concern.