Extended multi-stream temporal-attention module for skeleton-based human action recognition (HAR)

Abstract

Graph convolutional networks (GCNs) are an effective skeleton-based human action recognition (HAR) technique. GCNs enable the specification of CNNs to a non-Euclidean frame that is more flexible. The previous GCN-based models still have a lot of issues: (I) The graph structure is the same for all model layers and input data. GCN model's hierarchical structure and human action recognition input diversity make this a problematic approach; (II) Bone length and orientation are understudied due to their significance and variance in HAR. For this purpose, we introduce an Extended Multi-stream Temporal-attention Adaptive GCN (EMS-TAGCN). By training the network topology of the proposed model either consistently or independently according to the input data, this data-based technique makes graphs more flexible and faster to adapt to a new dataset. A spatial, temporal, and channel attention module helps the adaptive graph convolutional layer focus on joints, frames, and features. Hence, a multi-stream framework representing bones, joints, and their motion enhances recognition accuracy. Our proposed model outperforms the NTU RGBD for CS and CV by 0.6% and 1.4%, respectively, while Kinetics-skeleton Top-1 and Top-5 are 1.4% improved, UCF-101 has improved 2.34% accuracy and HMDB-51 dataset has significantly improved 1.8% accuracy. According to the results, our model has performed better than the other models. Our model consistently outperformed other models, and the results were statistically significant that demonstrating the superiority of our model for the task of HAR and its ability to provide the most reliable and accurate results.