ResFNN: Residual Structure-Based Feedforward Neural Network for Action Quality Assessment in Sports Consumer Electronics

Abstract

With the development of artificial intelligence (AI) and sports consumer electronics, AI-empowered Olympic sport technologies are being implemented more extensively. Action quality assessment (AQA), a sport action recognition and video refereeing technology, aims to automatically score action performance in videos obtained from sports consumer electronics deployed in arenas. It has gained much attention for its wide range of applications, such as sports event scoring, specific skill assessment, and rehabilitation medicine. General methods score action performance by directly regressing the initial video features to score, which neglects the possibility that the initial features are insufficiently effective. To address this issue, we propose a residual structure-based feedforward neural network (ResFNN) that enables efficient action feature learning to attain improved score assessment performance. First, the input videos are downsampled to clips and passed through inflated 3D convolutional networks (ConvNets) to obtain initial action video features. These features contain spatiotemporal information about the human actions occurring in the videos. Second, these features are aggregated and learned through our ResFNN. The ResFNN is composed of feedforward neural network residual blocks, which have strong function fitting and feature conversion capabilities. Therefore, the network learns features well and obtains more effective features. Third, a score distribution regression method is applied to obtain the underlying score distribution. This step establishes a more accurate mapping between the videos and scores. Finally, our method is demonstrated to outperform the majority of the existing methods through experiments conducted on the AQA-7, MTL-AQA, and JIGSAWS datasets.