Semantic information guided multimodal skeleton-based action recognition

Abstract

Human skeleton sequences are a crucial data modality for human motion representation. The primary challenge in skeleton-based action recognition lies in the effective capture of spatio-temporal correlations among skeleton joints. However, when the human body interacts with other objects in the background, these spatio-temporal correlations may become less apparent. To tackle this issue, we analyze the semantic information of human actions and propose a Semantic Information Guided Human Skeleton Action Recognition method (ActionGCL), which facilitates the differentiation of skeleton data from different action categories within a latent space. Concretely, we first construct a spatio-temporal action encoder based on graph convolutional neural networks to extract the dependencies among human skeleton sequences. It comprises alternating stacks of modules designed for temporal feature extraction and spatial graph convolution. The temporal feature extraction module integrates multiscale temporal convolutional networks to capture rich inter-frame correlations among nodes, while the spatial graph convolution module adaptively learns a sample-specific topology graph. Subsequently, to leverage the rich semantic information embedded within action labels, we design a multimodal contrastive learning module that simultaneously utilizes both skeleton and textual data. This module optimizes skeleton data in both skeleton-textual directions, employing the abundant semantic information within action labels to guide the training of spatio-temporal action encoders. It facilitates the accurate identification of ambiguous actions that are difficult to discern based solely on spatio-temporal correlations. Experimental results on two prominent action recognition datasets, NTU RGB+D 60 and NTU RGB+D 120, demonstrate that ActionGCL is effective and significantly outperforms other models in recognition accuracy.