SGCDiff: Sketch-Guided Cross-modal Diffusion Model for 3D shape completion

Abstract

Shape completion aims to generate complete shapes based on partial observations. Most recent methods utilize existing information on 3D shapes for shape completion tasks, such as inputting a partial 3D shape into an encoder–decoder structure to obtain a complete 3D shape. Despite the recent rapid evolution of neural networks greatly improving the completion performance of 3D shapes, they usually generate deterministic results. However, the completed shape is inherently diverse, leading to the concept of multimodal shape completion, in which a single partial shape can correspond to multiple plausible complete shapes. Existing multimodal shape completion methods are typically unpredictable, which results in the generated complete shapes exhibiting randomness. To address the challenge of achieving a guided generation process for multimodal shape completion, we propose a novel sketch-based diffusion model. Our key designs encompass the following. We propose a novel diffusion-based framework that employs sketches as guidance to generate complete 3D shapes. Within the framework, we introduce a dual cross-modal attention module that ensures the generated results retain sufficient geometric detail. Experimental results indicate that our approach not only facilitates multimodal shape completion based on sketches but also achieves competitive performance in deterministic shape completion.