Real-time facial reconstruction and expression replacement based on neural radiation field

Abstract

It is now possible to do high-fidelity 3D facial reconstruction and unique view synthesis thanks to the recent discovery of Neural Radiance Fields (NeRF), which has established its substantial importance in the field of 3D vision. However, the operational approaches that are now in use require a significant amount of human engagement, such as the need for users to provide semantic masks and the inconvenience of manual attribute searching for non-expert users. Our approach focuses on enabling the manipulation of NeRF-reconstructed faces with just a single text input. A scene manipulator, specifically a conditional version NeRF with deformable latent codes, is the first thing that this paper trains to accomplish this objective, in dynamic scenes, allowing facial deformations to be controlled through latent codes. However, to synthesize local deformations in a variety of contexts, it is not desirable to describe scene deformations using only a single latent coding. Therefore, this paper proposes a text-driven operation pipeline for facial reconstruction with NeRF, the development of an operating network that is capable of learning to represent scene changes using latent codes that vary at different spatial locations, and the integration of a WeChat mini-program to facilitate practical applications. This application approach enables even non-expert users to easily synthesize novel views. Our method has achieved a certain breakthrough in the field of 3D facial reconstruction, providing users with a simple and convenient text-driven operation approach.