PVSTrans: Patch-view-shape progressive interaction transformer for 3D shape recognition

Abstract

3D shape recognition has made substantial progress due to its wide-ranging applications and increasing research interest. Existing studies have investigated the paradigm of aggregating 3D shape descriptors derived from independently extracted view features. However, this stepwise approach has not fully capitalized on the intrinsic correlations between local regions of varying granularity and global shapes. To address this gap, we propose the Patch-View-Shape Progressive Interaction Transformer (PVSTrans), which enhances shape-patch interactions through progressive view-patch and shape-view interactions, effectively capturing essential dependencies among intra-view features, inter-view features, and global 3D shape features. Furthermore, by utilizing the byproducts of the progressive interaction process, specifically the attention weights of views and intra-view patches, we introduce a Shape-Guided Patch Selection strategy to dynamically identify significant patches in each view, which in conjunction with the multi-view features, forms a more informative 3D shape descriptors for final classification. Experimental results across diverse datasets, including ModelNet40, ScanObjectNN, FG3D, and ShapeNet Core55, demonstrate the effectiveness and generalizability of PVSTrans in 3D shape recognition tasks. Additionally, comprehensive experiments involving various views with differing quantities and spatial relations highlight the robustness of PVSTrans in handling incomplete views and irregular spatial configurations, showcasing its substantial potential for application in complex real-world scenarios. The code is available on https://github.com/Oli-lab-nun/PVSTrans.