Large Language Model-Based Spatio-Temporal Semantic Enhancement for Skeleton Action Understanding

Abstract

Skeleton-based temporal action segmentation aims to segment and classify human actions in untrimmed skeletal sequences. Existing methods struggle with distinguishing transition poses between adjacent frames and fail to adequately capture semantic dependencies between joints and actions. To address these challenges, we propose a large language model-based spatio-temporal semantic enhancement (LLM-STSE) method, a novel framework that combines adaptive spatio-temporal axial attention (ASTA-Attention) and dynamic semantic-guided multimodal action segmentation (DSG-MAS). ASTA-Attention models spatial and temporal dependencies using axial attention, whereas DSG-MAS dynamically generates semantic prompts based on joint motion and fuses them with skeleton features for more accurate segmentation. Experiments on MCFS and PKU-MMD datasets show that LLM-STSE achieves state-of-the-art performance, significantly improving action segmentation, especially in complex transitions, with substantial F1 score gains across multiple public datasets.