LAA: Local Awareness Attention for point cloud self-supervised representation learning

Abstract

Local awareness is essential for point cloud representation learning. In recent times, due to the increase in the amount of point cloud data and the success of the self-supervised learning paradigm in other domains, there has been an increase in the number of studies investigating point cloud self-supervised representation learning. However, the majority of current methods for implementing local awareness are incompatible with the paradigm of point cloud self-supervised pre-training, which makes it difficult for pre-trained models to benefit from it. Consequently, previous point cloud pre-training models have predominantly resulted in a global effective receptive field, with less focus on local awareness. A Gaussian-distributed, larger, more natural effective receptive field without artifacts will result in a superior representation of point cloud features. To address this issue, this paper proposes Local Awareness Attention (LAA), a plug-and-play module that enables local geometric perception while at the same time capturing global features. LAA consists of two branches. The first obtains local geometric information through the attention of each query and its neighborhood. The remaining branch learns global features through self-attention. The LAA module then fuses the features captured by the two branches through a single softmax, resulting in a competitive mechanism that achieves adaptive and multi-scale self-attention. Extensive experiments in indoor environments demonstrate that our LAA obtains stable effect enhancement in multiple transformer-based point cloud self-supervised pretraining networks, specifically outperforming multiple baselines by 0.1%–0.2% in ModelNet40 and by 0.2%–0.3% in ScanObjectNN.