A method for synthesizing dynamic image of virtual human

Abstract

With the rise of the Metaverse, the need for efficient modeling of avatars becomes increasingly urgent. Building virtual human models from human image datasets has been a hot topic in computer vision. We used the speech synthesis technology to complete the conversion from text to speech waveform, and used the speech-lip shape generation method to generate a real person image with audio and video synchronization, finally used the thin plate spline transformation method to drive the virtual human image, and synthesizes a virtual human with audio and video synchronization image. Experimental results show that this method can effectively solve the problem of text-driven avatar lip mismatch and text-driven avatar audio and video asynchronous problems, and can synthesize high-quality, high-fidelity, low-latency avatars.