SiTrEx: Siamese transformer for feedback and posture correction on workout exercises

Applying Machine Learning and Deep Learning techniques to sequences of Human Pose Landmarks to recognize workout exercises and count repetitions is widely studied in the computer vision literature. However, existing approaches suffer from two major problems. The first issue is that they lack the ability to provide detailed feedback on the postures performed by the athletes or provide feedback for a limited range of exercises using hand-designed rules and algorithms. The second problem is that these approaches consider only a predefined set of exercises and do not generalize to exercises outside their training data, which limits their usability. In this paper, we aim to address these two shortcomings by proposing a one-shot learning approach that utilizes Siamese Transformers to provide detailed feedback on individual human joints and can generalize to new exercises that are not present in the used dataset. The proposed configuration of the Siamese Transformer model deviates from its standard use in that it outputs a vector of similarity indicators rather than a single similarity score. Additionally, an accompanying binary classification Transformer model is used to assess the usefulness of different parts of the human pose for the input exercise without prior knowledge of the exercise itself. These properties allow the proposed approach to be used in general-purpose fitness applications and coach/athlete training platforms. The proposed approach achieved a 5-fold cross-validation test accuracy of $94.4\%\pm 0.8$ on the collected dataset.