A cross-modal Siamese representation learning network for point cloud understanding

Abstract

Learning effective representations from unannotated point cloud data is a challenging task in self-supervised learning. Recently, methods that use point clouds and images for cross-modal learning have achieved impressive performance. However, these methods still have some shortcomings in exploring the latent information between these two modalities. To address this issue, we propose a cross-modal Siamese representation learning network called CrossSiamese. This network uses point clouds and their rendered images for cross-modal contrastive learning. We introduce an intra-modal prediction mechanism in the network to capture the internal information in the point cloud and image modalities. In addition, we introduce a cross-modal cross-prediction mechanism to capture mutual information between the two modalities. Experimental results show that our method improves the accuracy of linear classification for 3D objects by 0.4% on ModelNet40 and 1.7% on ScanObjectNN compared to existing baseline methods. Additionally, experiments on few-shot object classification and 3D object part segmentation further validate the effectiveness of our method. These results indicate that the representations learned by our method have generalization ability and can be effectively transferred to these three downstream tasks.