Sample-based synthesis of talking heads

Abstract

Synthesizing photo-realistic talking heads is a challenging problem, and so far all attempts using conventional computer graphics produced heads with a distinctly synthetic look. In order to look credible, a head must show a picture-perfect appearance, natural head movements, and good lip-sound synchronization. We use sample-based graphics to achieve more photo-realistic appearances than what is possible with the traditional approach of 3D modeling and texture mapping. For sample-based graphics, first parts of faces are cut from recorded images and are scored in a database. New sequences are then synthesized by integrating such parts into whole faces. With sufficient recorded data this approach produces by far the most naturally looking speech articulation. We integrate 3D modeling with the sample-based technique in order to enhance its flexibility. This allows, for example, showing the head over a much wider range of orientations.