Generation of realistic facial animation of a CG avatar speaking a moraic language

Abstract

We propose a new method for generating real-time realistic facial animation using face mesh data corresponding to the fifty-six C+V (Consonant and Vowel) type morae that form the basis of Japanese speech. This method produces facial expressions by weighted addition of fifty-three face meshes based on the mapping of voice streaming to registered morae in real-time. Both photogrammetric models and existing off-the-shelf head models can be used as face meshes. Natural facial expressions of speech can be synthesized from a modeling to live animation in less than two hours. The results of a user study of our method showed that the facial expression during Japanese speech was more natural than popular real-time methods to generate facial animation, English-base Oculus Lipsync and volume intensity based facial animations.