HiSURF: Hierarchical semantic-guided unified radiance field for generalizing across unseen scenes

Abstract

Recent advancements in neural field representations have significantly improved novel view synthesis for seen scenes. However, generalizing seen representations to unseen scenes remains challenging. Addressing this problem, we propose the Hierarchical Semantic-guided Unified Radiance Field (HiSURF) to leverage hierarchical semantic attributes from seen scenes as prior knowledge. The synthesis of scene representations for unseen environments can be enabled by establishing an interpretable mapping between semantic attributes and visual features. Specifically, HiSURF consists of a local semantic embedding module, a global semantic mapping module, and a composite rendering module. For a scene with multiple objects, the local module disentangles attributes of objects to generate fine object-level triplanes, which preserve structural and surface details for objects. At the same time, the global module utilizes attributes of the holistic scene to construct a coarse scene-level triplane, which ensures layout consistency and contextual coherence for the scene. Then, the composite rendering module integrates features from both object-level and scene-level triplanes for high-quality novel view synthesis. Experimental results on the ClevrTex and Kubric datasets demonstrate that our HiSURF not only outperforms existing approaches in novel view synthesis but also exhibits superior generalization capability to unseen scenes.