DESTalker: Disentangling Emotion and Style for Expressive 3D Facial Animation via Residual Generation

Abstract

Speech-driven 3D facial animation has achieved remarkable lip-synchronization. However, generating expressive and personalized animations remains a formidable challenge. Existing methods often struggle with the entanglement of emotion and speaking style, as well as semantic conflicts arising from multi-modal inputs. To address these limitations, we propose DESTalker, a framework that disentangles emotion and style to generate expressive facial motion via residual modulation. We introduce an information-theoretic disentanglement strategy utilizing mutual information minimization and orthogonal constraints to robustly separate audio content, motion emotion, and speaking style. Furthermore, we adopt a residual generation strategy where neutral lip-sync motion is synthesized first, followed by the injection of emotion and style as residual components to preserve articulatory precision. Finally, to resolve cross-modal conflicts, we design a semantic consistency-guided soft-gating mechanism that dynamically regulates visual emotion integration under audio dominance. Extensive experiments on the 3DMEAD dataset demonstrate that DESTalker outperforms state-of-the-art methods in emotional expressiveness, style consistency, and lip-synchronization accuracy.