Sequential selection and calibration of video frames for 3D outdoor scene reconstruction

Abstract

3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene, including the scene layout, objects bounding boxes and shapes. Existing holistic scene understanding methods primarily recover scenes from single images, with a focus on indoor scenes. Due to the complexity of real-world, the information provided by a single image is limited, resulting in issues such as object occlusion and omission. Furthermore, captured data from outdoor scenes exhibits characteristics of sparsity, strong temporal dependencies and a lack of annotations. Consequently, the task of understanding and reconstructing outdoor scenes is highly challenging. The authors propose a sparse multi-view images-based 3D scene reconstruction framework (SMSR). It divides the scene reconstruction task into three stages: initial prediction, refinement, and fusion stage. The first two stages extract 3D scene representations from each viewpoint, while the final stage involves selection, calibration and fusion of object positions and orientations across different viewpoints. SMSR effectively address the issue of object omission by utilizing small-scale sequential scene information. Experimental results on the general outdoor scene dataset UrbanScene3D-Art Sci and our proprietary dataset Software College Aerial Time-series Images, demonstrate that SMSR achieves superior performance in the scene understanding and reconstruction.