Text-based Talking Facial Synthesis for Virtual Host System

Abstract

With the prevailing of deep learning technology, automatic virtual image synthesis has made huge progress and the popularity of virtual portraits has been growing rapidly. Traditional virtual synthesis system rely on computer graphics method driven by motion capture of a real person, which need labor and equipment costs. In view of this, our paper proposes a virtual host synthesis method based on text driven to generate lip shape and facial animation(include eye movement and head pose) from a signal facial image of a virtual host. More precisely, we use three main modules to construct a virtual host synthesis system: a speech synthesis module based on Tacotron2, a speech to landmark points module to extract mixture landmarks movement to speech, and video generation module based on conditional generative adversarial network to generate video frames and realize time-continuous automatic sport news reporting.