Rotation invariant dual-view 3D point cloud reconstruction with geometrical consistency based feature aggregation

Abstract

Multi-view 3D reconstruction usually aggregates the features from object with different views for recovering 3D shape. We argue that exploring the rotation invariance of object region and further learning the geometrical consistency of regions across views enables better feature aggregation. However, existing methods fail to investigate this insight. Meanwhile, the intrinsic self-occlusion existed in input views would also compromise the consistency learning. This paper presents an approach termed Rotation invariant dual-view 3D point cloud reconstruction with Geometrical consistency based Feature aggregation (R3GF), reconstructing a 3D point cloud from two RGB images with arbitrary views. In encoding, a point cloud initialization network is assigned to initialize a rough point cloud for each view. To explore the rotation invariance of object region, a regional feature extraction network is proposed. It uses Euclidean distance and angle-based clues to capture rotation-invariant features that characterize geometrical information from different regions of rough point clouds. In decoding, to perform consistency learning even when self-occlusion existed in input views, a dual-stage cross attention mechanism is devised. It enhances the captured regional features by global shapes of rough point clouds, enriching the information of occluded regions. Then, the enhanced regional features from rough point clouds with different views are aligned to model the geometrical consistency among regions, achieving feature aggregation accurately. Furthermore, a point cloud refinement module is constructed to produce a refined point cloud using the aggregated feature. Extensive experiments on the ShapeNet and Pix3D datasets show that our R3GF outperforms the state-of-the-art methods.