3D modeling from a single sketch with multifaceted semantic understanding

Abstract

This paper studies the problem of 3D shape generation from a single sketch. Prior works rely on directly extracted visual features of sketches as guidance for the generation process. However, the sparse visual cues and abstract nature of sketches, which are inherited in the guiding features, lead to semantic ambiguity and geometry incompleteness in the generated shapes, compromising accuracy. To address this, we propose MSU-3D, a diffusion-based framework for sketch-to-3D generation, leveraging Multifaceted Semantic Understanding to explicitly analyze the construction information of sketches from multiple facets before providing fine-grained guidance over 3D shape generation. Specifically, we decompose sketches through three interpretative facets (semantics, depth, and normal), introducing reasoning of three representations to capture 3D features from distinct perspectives: local components, basic 3D geometry, and 3D surface details. One step further, we propose a multifaceted perception module. It aggregates multifaceted feature representations and leverages local component features as a two-pronged guiding representation to jointly guide the perception of basic shapes and surface details. To ensure fine-grained control, the hierarchical perception strategy adaptively injects varying granularity of perception features at different stages of the 3D generation. Extensive experiments and comparisons with state-of-the-art methods on various complex posture datasets validate the effectiveness of our framework in mitigating semantic ambiguity and geometry incompleteness in 3D generation.