Mosaic: Towards Efficient and Low-Latency Multi-NeRF Complex Scene Cloud Rendering

Abstract

Neural Radiance Fields (NeRF) offers superior reconstruction accuracy over traditional methods like point clouds. This is particularly advantageous for applications in retail, VR, and navigation. However, when representing a complex scene with multiple semantically meaningful objects, especially those containing high-frequency details, the conventional single NeRF approach struggles to accurately capture these intricate details. Furthermore, the state-of-the-art multi-NeRF solutions, though effectively capturing details by representing each object as a separate NeRF, are too resource-intensive to be run on mobile devices. Existing NeRF rendering acceleration studies focus on single NeRF data representation and its framework improvements, without resorting to the visibility information for further optimization and considering the practical multi-NeRF rendering scenario. In this paper, we present Mosaic, a multi-NeRF cloud rendering system that facilitates high-fidelity, and low-latency cloud rendering for complex scenes. We first design a multi-resolution controller to dynamically determine the resolution for each pixel based on its contextual details and view direction. It then incorporates a visibility-aware region pruner to effectively eliminate unneeded pixels to avoid being rendered. The extensive experiments on various scenes show that Mosaic can achieve high-fidelity rendering with 43% lower transmission cost and up to 70% smaller computation consumption when compared with other state-of-the-art benchmarks.