Generating Talking Facial Videos Driven by Speech Using 3D Model and Motion Model

Abstract

Facial expression is one of the most important features of a face. In previous works on generating talking facial videos driven by speech without additional driving information, existing models struggled to directly learn the mapping from speech to facial features, resulting in poor quality of generated facial expressions. In this paper, we propose a method for generating speech-driven facial videos using 3D models and motion capture. This method demonstrates good performance in terms of model robustness, adaptation to large head poses, and improvement of fine-grained facial expression details. We learn features from speech, reconstruct the face by fitting 3DMM coefficients using speech, and employ a motion-captured based generative adversarial network to ensure clear facial texture details in the generated faces. On the publicly available dataset VoxCeleb2, our method achieves scores of 31.22 in PSNR, 0.89 in SSIM, 19.4 in FID, and 1.96 in F_LMD, outperforming other methods. On the MEAD dataset, our method achieves scores of 30.65 in PSNR, 0.68 in SSIM, 20.5 in FID, 2.36 in SyncNet, and 2.45 in F_LMD, outperforming other methods. Experimental results demonstrate that our method effectively enhances the model robustness for speech-driven facial video generation without additional driving information.