SceneFactory: A Workflow-Centric and Unified Framework for Incremental Scene Modeling

Abstract

In this article, we present SceneFactory, a workflow-centric and unified framework for incremental scene modeling that conveniently supports a wide range of applications, such as (unposed and/or uncalibrated) multiview depth estimation, LiDAR completion, (dense) RGB-D/RGB-LiDAR (RGB-L)/Mono/Depth-only reconstruction, and simultaneous localization and mapping (SLAM). The workflow-centric design uses multiple blocks as the basis for constructing different production lines. The supported applications, i.e., productions avoid redundancy in their designs. Thus, the focus is placed on each block itself for independent expansion. To support all input combinations, our implementation consists of four building blocks that form SceneFactory: first, tracking, second, flexion, third, depth estimation, and fourth, scene reconstruction. The tracking block is based on Mono SLAM and is extended to support RGB-D and RGB-L inputs. Flexion is used to convert the depth image (untrackable) into a trackable image. For general-purpose depth estimation, we propose an unposed and uncalibrated multiview depth estimation model (U$^{2}$-MVD) to estimate dense geometry. U$^{2}$-MVD exploits dense bundle adjustment to solve for poses, intrinsics, and inverse depth. A semantic-aware ScaleCov step is then introduced to complete the multiview depth. Relying on U$^{2}$-MVD, SceneFactory both supports user-friendly 3-D creation (with just images) and bridges the applications of Dense RGB-D and Dense Mono. For high-quality surface and color reconstruction, we propose dual-purpose multiresolutional neural points for the first surface accessible surface color field design, where we introduce improved point rasterization for point cloud-based surface query. We implement and experiment with SceneFactory to demonstrate its broad applicability and high flexibility. Its quality also competes or exceeds the tightly-coupled state of the art approaches in all tasks.