3D shape analysis via multi-modal contrastive learning

Abstract

In recent years, 3D shape analysis has emerged as a crucial field with applications in various domains, such as multimedia processing, computer graphics, computer vision, and robotics. The ability to understand and interpret 3D shapes is fundamental for tasks like 3D shape segmentation, points of interest detection, shape retrieval, recognition, and generation. However, the complexity of 3D mesh models is a significant barrier that stops the topic from enhancing. Thus, we propose a novel 3D shape analysis framework in this paper by multi-modal contrastive learning techniques. Our framework makes use of the original mesh data and the projected images from various points of view of the mesh model. Those two modals contribute to providing more precise features with the help of our within-modal and cross-modal losses, which respectively calculate the distances of feature vectors within the mesh model and between feature vectors of mesh and image. Our framework is tested on downstream tasks, including 3D shape segmentation and points of interest detection, and outperforms most state-of-the-art methods on public datasets.