3DGM: Deformable and Texturable 3D Gaussian Model via Level-of-Detail Proxy

Abstract

3D Gaussian Splatting has markedly impacted neural rendering by achieving impressive fidelity and performance. Despite this achievement, it is not readily applicable to developing interactive applications. Real-time applications like XR apps and games require functions such as animation, UV mapping and level of detail (LOD) simultaneously manipulated through a 3D model. To address this need, we propose a modelling strategy analogous to typical 3D models, which we call 3D Gaussian Model (3DGM). 3DGM relies on attaching 3D Gaussians on the triangles of a mesh proxy, and the key idea is to bind sheared 3D Gaussians in texture space and re-projecting them back to world space through implicit shell mapping; this design naturally enables deformation and UV mapping via the proxy. Further, to optimize speed and fidelity based on different viewing distances, each triangle can be tessellated to change the number of involved 3D Gaussians adaptively. Application-wise, we will show that our proxy-based 3DGM is capable of enabling novel deformation without animated training data, texture transferring via UV mapping of the 3D Gaussians, and LOD rendering. The results indicate that our model achieves better fidelity for deformation and better optimization of fidelity and performance given different viewing distances. Further, we believe the results indicate the potential of our work for enabling interactive applications for 3D Gaussian Splatting.