Multimodality Consistency for Point Cloud Completion via Differentiable Rendering

Abstract

Point cloud completion aims to acquire complete and high-fidelity point clouds from partial and low-quality point clouds, which are used in remote sensing applications. Existing methods tend to solve this problem solely from the point cloud modality, limiting the completion process to only 3-D structure while overlooking the information from other modalities. Nevertheless, additional modalities possess valuable information that can greatly enhance the effectiveness of point cloud completion. The edge information in depth images can serve as a supervisory signal for ensuring accurate outlines and overall shape. To this end, we propose a brand-new point cloud completion network, dubbed multimodality differentiable rendering (MMDR), which utilizes point-based differentiable rendering (DR) to obtain the depth images to ensure that the model preserves the point cloud structures from the depth image domain. Moreover, the attentional feature extractor (AFE) module is devised to exploit the global features inherent in the partial input, and the extracted global features together with the coordinates and features of the patch center are fed into the point roots predictor (PRP) module to obtain a set of point roots for the upsampling module with point upsampling Transformer (PU-Transformer). Furthermore, the multimodality consistency loss between the depth images from predicted point clouds and corresponding ground truth enables the PU-Transformer to generate a high-fidelity point cloud with predicted point agents. Extensive experiments conducted on various existing datasets give evidence that MMDR surpasses the off-the-shelf methods for point cloud completion after qualitative and quantitative analysis.