Causality-inspired representation learning for weakly supervised skeleton-based action recognition

Skeleton-based action recognition has become an important research topic in computer vision. Most existing methods tackle this task by mapping skeletal features to fine-grained labels. However, weakly labeled samples—containing both relevant and irrelevant human instances—inevitably emerge during data collection and annotation, adversely impacting model training and inference. To address this challenge, we utilize a structural causal model to formalize the Weakly Supervised Skeleton-based Action Recognition (WS-SAR) problem, treating relevant and irrelevant instances as causal and non-causal factors, respectively. Our primary goal is to learn representations of relevant human instances, i.e., causal factors, from weakly labeled data in WS-SAR. Based on this formulation, we propose a Causality-inspired Representation Learning (CiRL) framework, comprising the Causality DEtection TRansformer (C-DETR) and Supervised Contrastive Learning (Sup-CL). C-DETR leverages learned embeddings as class queries and employs class-matching along with causality-enhanced contrastive learning to extract causal factors from both sample-level and instance-level features. Subsequently, the Sup-CL training strategy applies supervised contrastive learning to effectively capture shared causal representations among weakly labeled samples within the same class. Experimental results show that our framework achieves state-of-the-art performance across multiple datasets, including WL-NTU, IT-NTU120, and SBU. The source code is available at https://github.com/KennCoder7/CiRL.