SG-CLR: Semantic representation-guided contrastive learning for self-supervised skeleton-based action recognition

Abstract

Contrastive learning and multimodal representation learning have been widely applied to skeleton-based action recognition. However, the majority of the research focuses on the mining of spatial- temporal features while ignoring the semantic information of action. To deal with these drawbacks, we propose a novel contrastive learning framework (SG-CLR) for skeleton-based action recognition, which captures fine-grained multi-level discriminative features by incorporating both semantic compensation and spatial–temporal feature reinforcement. For semantic compensation contrastive learning, in order to achieve dynamic compensation of high-order semantic information, combining LLMs-generated action descriptions with multi-modal encoders to integrate cross-modal multivariate features (e.g., skeleton and text features). For spatial–temporal enhancement contrastive learning, SkeleMask augmentation is proposed to mine more high-level temporal movement information. Experiments demonstrate that the proposed SG-CLR achieves the state-of-the-art performance on the NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets. Related code will be available at https://github.com/QingZhiWMY/SG-CLR.