Image-based view-dependent appearance for 3D Gaussian splatting

Abstract

3D Gaussian Splatting (3DGS) has achieved significant progress in the field of novel view synthesis. However, there are challenges associated with using spherical harmonics to learn scenes that involve specular reflections, due to the presence of high-frequency details in such scenes. To address above problem, we propose an image-based view-dependent appearance model to jointly extracts both high- and low-frequency information from the scene, to more efficiently represent the appearance field of 3D Gaussians. Specifically, by statistically assessing the dot product between the view direction and the normal at the respective Gaussian within the image, we develop a view-dependent appearance module that calculates the variances of these dot products; the module is able to adaptively assign weights to both specular and diffuse reflection colors. We propose a normal-guided specular reflections module to extract view-dependent high-frequency information, which effectively filters out specular colors by using a threshold on the variance of the dot product between the view direction and the normal. In addition, to extract low-frequency information, we design an image-based diffuse reflections module to compute the diffuse reflection colors and preserve full-frequency information. Experimental results show that our method outperforms the baseline in both quantitative and qualitative results, significantly enhancing the ability of 3DGS in processing specular reflection scenes.