MMGS: Multi-Model Synergistic Gaussian Splatting for Sparse View Synthesis

Abstract

3D Gaussian Splatting (3DGS) generates a field composed of 3D Gaussians to represent a scene. As the number of input training views decreases, the range of possible solutions that fit only training views expands significantly, making it challenging to identify the optimal result for 3DGS. To this end, a synergistic method is proposed during training and rendering under sparse inputs. The proposed method consists of two main components: Synergistic Transition and Synergistic Rendering. During training, we utilize multiple Gaussian fields to synergize their contributions and determine whether each Gaussian primitive has fallen into an ambiguous region. These regions impede the process for Gaussian primitives to discover alternative positions. This work extends Stochastic Gradient Langevin Dynamic updating and proposes a reformulated version of it. With this reformulation, the Gaussian primitives stuck in ambiguous regions adjust their positions, enabling them to explore an alternative solution. Furthermore, a Synergistic Rendering strategy is implemented during the rendering process. With Gaussian fields trained in the first stage, this approach synergizes the parallel branches to improve the quality of the rendered outputs. With Synergistic Transition and Synergistic Rendering, our method achieves photo-realistic novel view synthesis results under sparse inputs. Extensive experiments demonstrate that our method outperforms previous methods across diverse datasets, including LLFF, Mip-NeRF360, and Blender.