Controlling Eye Blink for Talking Face Generation via Eye Conversion

Abstract

A real talking face video includes not only the movement of the mouth, but also realistic blinking details. For a computer generated talking face video, realistic eye movements are critical to overcome the uncanny valley effect. However, it remains a great challenge to introduce realistic eye movements into talking face generation systems. In this paper, we propose a two-stage system for generating talking face video with realistic controllable blinking actions. Through eye conversion and frame replacement, our architecture can ensure the controllability of the blinking motion generation. We propose an eye conversion GAN, which can convert a face image into any stages of blinking, and maintain the consistency of facial identity features. In this network, we design joint training to increase the network’s ability of generating closed and half-closed eye images, which improves the authenticity of the eyes. Experiments on two popular data sets show that compared with previous work, our method can not only guarantee the authenticity of mouth movements, but also generate realistic and controllable eye blinks.