MTSegNet: Manifold Transformer for 3D shape segmentation

Abstract

The semantic segmentation of 3D meshes is a critical component of 3D shape analysis, which involves assigning semantic labels to each face of a 3D mesh. Despite its significance, current methods often struggle to capture manifold information in 3D meshes, a fundamental characteristic distinguishing them from other representation forms of 3D data, like 3D point clouds or 3D voxels, resulting in suboptimal segmentation outcomes. In this paper, we propose a novel Transformer-based approach, Manifold Transformer (MTSegNet), for 3D mesh semantic segmentation, which effectively learns manifold information. By using hierarchical Transformers, MTSegNet can capture both local and global features of 3D meshes, while reducing the computational complexity and memory consumption. To further improve the performance of our method, we design an effective input-generating algorithm that serializes input data into multiple sequences of tokens that represent the geometry and topology of 3D meshes. This algorithm preserves the structural information and spatial relations of 3D meshes, while enabling the use of standard Transformer architectures. The proposed method is evaluated on four benchmark datasets: PSB, COSEG, ShapeNetCore, and HumanBody, and it achieves state-of-the-art results on all datasets, outperforming the previous methods.