UV Gaussians: Joint learning of mesh deformation and Gaussian textures for human avatar modeling

Abstract

Reconstructing photo-realistic drivable human avatars from multi-view image sequences has been a popular and challenging topic in the field of computer vision and graphics. While existing NeRF-based methods can achieve high-quality novel view rendering of human models, both training and inference processes are time-consuming. Recent approaches have utilized 3D Gaussians to represent the human body, enabling faster training and rendering. However, they undermine the importance of the mesh guidance and directly predict Gaussians in 3D space with coarse mesh guidance. This hinders the learning procedure of the Gaussians and tends to produce blurry textures. Therefore, this paper proposes UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. The method utilizes the embedding of UV map to learn Gaussian textures in 2D space, leveraging the capabilities of powerful 2D networks to extract features. Additionally, through an independent Mesh network, the approach optimizes pose-dependent geometric deformations, thereby guiding Gaussian rendering and significantly enhancing rendering quality. A new dataset of human motion has been collected and processed, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that the proposed method achieves state-of-the-art synthesis of novel view and novel pose. The code and data will be made available as open-source.