PHMG: Prompt-based Human Motion Generation for action recognition

Data generation is an effective method to address inefficient and costly data collection in action recognition. Skeleton data is more robust to illumination and background than RGB data. Therefore, the generation of skeleton motions holds greater value. Existing skeleton motion generation methods generate motions that deviate from the real motion data distribution, leading to blurred inter-class boundaries and adversely affecting action recognition accuracy. In this paper, we propose a Prompt-based Human Motion Generation Network (PHMG), which consists of a Prompt-based Generation Module (PGM) and an Active Optimization Module (AOM). The encoder within the PGM integrates spatio-temporal dual-branch self-attention with graph convolution, effectively capturing both local and global motion features while maintaining the independence of spatio-temporal representations. Moreover, the PGM integrates Contrastive Language–Image Pre-Training (CLIP) encoded textual prompts into the generation process adaptively through the proposed Adaptive Weight(AW). The AOM comprises a recognition network and an active optimization layer. The recognition network produces prediction vectors for the motions generated by the PGM, while the active optimization layer evaluates these vectors using an uncertainty metric to optimize the generated motions. The PGM and AOM operate alternately to generate a refined set of motions iteratively. Extensive experiments on public datasets, namely NTU-RGB+D and NTU-RGB+D 120, reveals that our PHMG achieves excellent results in both qualitative and quantitative assessments. Notably, we attain 2.48 FMD, 92.98% accuracy on NTU-RGB+D, and 9.24 FMD, 58.47% accuracy on NTU-RGB+D 120.