Speech-assisted facial expression analysis and synthesis for virtual conferencing systems

Fast, reliable, and marker-free facial expression analysis still remains to be a difficult task in computer vision research. In this paper, the concept of speech-assisted facial expression analysis and synthesis is proposed, which shows that the speech-driven facial animation technique not only can be used for expression analysis. From the input speech, the mouth shape can is estimated from the audio-visual model. Thus, the large search space of mouth appearance is reduced for mouth tracking. Similarly, the modeling technique is extended from modeling speech and mouth shape to facial movements and detail facial texture changes. In this way, a virtual conferencing system with video realistic avatars is realized to meet real-time requirement.