EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion

Abstract

The task of audio-driven portrait animation involves generating a talking head video using an identity image and an audio track of speech. While many existing approaches focus on lip synchronization and video quality, few tackle the challenge of generating emotional talking head videos. The ability to control emotions is essential for producing expressive and realistic animations. To solve this problem, a novel diffusion-based emotional talking head generation method (EMOdiffhead) is presented that not only enables fine-grained control of emotion categories and intensities but also enables one-shot generation. Given the faces learned with an articulated model and expression (FLAME) 3D model’s linearity in expression modeling, the Flame-based detailed expression capture and animation (DECA) method is employed to extract expression vectors that are combined with audio to guide a diffusion model in generating videos with precise lip synchronization and rich emotional expressiveness. EMOdiffhead enables learning rich facial information from emotion-irrelevant data, thus facilitating emotional video generation. It effectively overcomes the limitations of emotional data, such as the lack of diversity in facial and background information, and addresses the absence of emotional details in emotion-irrelevant data. Extensive experiments and user studies demonstrate that the proposed method achieves state-of-the-art performance compared to other emotional portrait animation methods.