Mastering Fencing Techniques with Machine Learning: A Video-Based Classification and Correction System

Abstract

In fencing and other sports, athletes must continually execute numerous movements absent of expert supervision. In this article, we address this issue by creating a smart coach that classffies and corrects fencing movements using video footage. To avoid contextual bias, a variety of machine learning models, including LSTM, CNN-BiLSTM and attention based models were trained on fencers’ keypoints. For this purpose, we collected and annotated a video dataset featuring more than 1200 videos of four fencing movements and corresponding error patterns. This will be published along with this work under the Creative Commons License 4.0. The actions in this dataset can be classified with the masked self-attention architecture attaining a macro-averaged F1 score of over 99% on the test set. In an additional effort to show the impact of label quality, the performances are boosted on average by 3% with the introduction of multi-labeling.