I3D-AE-LSTM: Combining action representations using a 2-stream autoencoder for Action Quality Assessment

Abstract

Systems dedicated to the Action Quality Assessment (AQA) have seen a notable surge in interest from both scholars and industry experts. Such systems seek to provide an objective measure of the quality of athletes’ physical movements, offering perspectives that were once exclusive to skilled human evaluators. Our research builds upon the observation that previous studies have primarily considered spatio-temporal features and pose estimation keypoints as distinct elements within action analysis. We introduce an innovative two-stream methodology that merges these representations of an action. Our approach leverages the combined capabilities of various techniques: utilising Inflated 3D ConvNet (I3D) for the extraction of spatial–temporal characteristics from video content, employing OpenPose for detailed pose estimation keypoints that furnish an Autoencoder (AE) with intricate information to concentrate on key aspects of action, and employing Long Short Term Memory (LSTM) networks. The integration of the proposed two-stream methodology allows for a more comprehensive analysis of athletic movements. By capturing both spatial and temporal dynamics together, we can better understand the nuances of an action. Traditional methods often treat these aspects separately, leading to a fragmented understanding. In addition, incorporating detailed pose estimation through OpenPose allows the Autoencoder to focus on key aspects of the action. This targeted focus ensures that the evaluation is not only about how movements look in a general sense but also how they align with optimal performance criteria. By combining the representations, we enhance the model’s ability to recognise and evaluate complex movements more accurately. Furthermore, we propose a new multi-variate scoring system designed to assess action quality based on the scores from individual judges. Multi-variate scoring introduces a richer dataset for assessing action quality, enabling more sophisticated analyses and decision-making. Multi-variate scoring can better highlight discrepancies or consensus among judges, leading to more reliable assessments. The method has an average Spearman Rank Correlation of 97.35%, which outperforms current state-of-the-art methods and underscores the effectiveness of merging spatio-temporal and pose estimation keypoints into a unified action representation. Finally, training the model to predict the scores assigned by each judge reveals additional advantages. In sports involving multiple scoring criteria, the proposed approach enables the extraction of more detailed insights, marking a significant advancement that allows for more precise and detailed predictions of scores.