Rugularizing generalizable neural radiance field with limited-view images

Abstract

We present a novel learning model with attention and prior guidance for view synthesis. In contrast to previous works that focus on optimizing for specific scenes with densely captured views, our model explores a generic deep neural framework to reconstruct radiance fields from a limited number of input views. To address challenges arising from under-constrained conditions, our approach employs cost volumes for geometry-aware scene reasoning, and integrates relevant knowledge from the ray-cast space and the surrounding-view space using an attention model. Additionally, a denoising diffusion model learns a prior over scene color, facilitating regularization of the training process and enabling high-quality radiance field reconstruction. Experimental results on diverse benchmark datasets demonstrate that our approach can generalize across scenes and produce realistic view synthesis results using only three input images, surpassing the performance of previous state-of-the-art methods. Moreover, our reconstructed radiance field can be effectively optimized by fine-tuning the target scene to achieve higher quality results with reduced optimization time. The code will be released at https://github.com/dsdefv/nerf.