Multi-modal semantic embedding network for 3D shape recognition and retrieval

Abstract

Current methods for 3D shape recognition and retrieval utilize deep learning techniques, achieving commendable performance through a singular representation while neglecting the multi-modal information inherent to the same 3D object. Furthermore, certain approaches treat recognition and retrieval as distinct tasks; however, these processes should be synergistic rather than antagonistic. In this paper, we propose a multi-modal semantic embedding network designed to deliver a more comprehensive representation of 3D shapes, thereby enhancing recognition accuracy and retrieval efficacy. Initially, we employ two independent feature extractors to derive multi-view and point cloud features. Subsequently, we introduce a multi-modal feature fusion method that emphasizes uncovering correlations between diverse modal features while mitigating information degradation. Finally, we implement a joint learning strategy for the fused features that resolves modal heterogeneity and facilitates joint mapping of visual attributes with semantic labels. Extensive experiments on multiple datasets validate the superiority of our approach.