Semantic-aware hierarchical clustering for inverse rendering in indoor scenes

Abstract

Decomposing a scene into its material properties and illumination, given the geometry and multi-view HDR observations of an indoor environment, is a fundamental yet challenging problem in computer vision and graphics. Existing approaches, combined with neural rendering techniques, have shown promising results in object-specific scenarios but often struggle with inconsistencies in material estimation within complex indoor scenes. Besides, ambiguities frequently arise between lighting and material properties. To address these limitations, we propose an adaptive inverse rendering pipeline based on Factorized Inverse Path Tracing (FIPT) that incorporates a semantic-aware hierarchical clustering approach. This enhancement enables the disentanglement of lighting and material properties, facilitating more accurate and consistent estimations of albedo, roughness, and metallic characteristics. Additionally, we introduce a voxel grid filter to further reduce computational time. Experimental results on both synthetic and real-world room-scale scenes demonstrate that our method produces more accurate material estimations compared to state-of-the-art methods. Furthermore, we demonstrate the potential of our method through several applications, including novel view synthesis, object insertion, and relighting.