Realistic Facial Expression Reconstruction Using Millimeter Wave

Abstract

The technology of facial expression reconstruction has paved the way for various face-centric applications such as virtual reality (VR) modeling, human-computer interaction, and affective computing. Existing vision-based solutions present challenges in privacy leakage and poor lighting conditions. In this paper, we introduce a nonintrusive facial expression reconstruction system, mm3DFace, which uses a millimeter wave (mmWave) radar to reconstruct facial expressions in a privacy-preserving and passive manner. mm3DFace first captures and pre-processes mmWave signals reflected by a human face, and extracts intricate facial geometric features using a ConvNeXt model integrated with triple loss embedding. Subsequently, mm3DFace derives pose-invariant facial representations utilizing region-divided affine transformation, and further generates individual facial shapes with 68 facial landmarks. Then, dynamic facial expressions with 3D facial avatars are reconstructed to exhibit realistic facial expressions. Finally, mm3DFace enables micro-expression recognition with mmWave signals, which ensures the capability of describing tiny facial changes. Through extensive real-world experiments involving 15 participants, mm3DFace achieves a normalized mean error of 3.94%, a mean absolute error of 2.30 mm, and a 3D-mean absolute error of 4.10 mm in tracking 68 facial landmarks, which demonstrates the efficacy and practicality of mm3DFace in real-world 3D facial reconstruction scenarios.