Self-adaptive uncertainty modeling and relation reasoning for cross-modal egocentric human action recognition in sports

Human action recognition (HAR) aims to interpret human actions from provided data and has found widespread application in the sports domain. However, HAR in sports presents unique challenges, as egocentric video often lacks complete body information. Additionally, individual variations and aleatoric errors during data collection can further corrupt cross-modal interactions. For example, the movement habits of different athletes, differences in body types, and changes in the wearing positions of sensors (such as IMUs) can lead to individual differences and random noise in inertial data. To address these issues, we emphasize the distinctiveness of human actions in sports and propose a novel framework for robust feature embedding and enhanced cross-modal interaction, termed Self-adaptive Uncertainty Modeling and Relation Reasoning (SUMRR), specifically designed for egocentric human action recognition in sports. Our approach begins with the sampling of robust unimodal features from an uncertainty perspective, which helps mitigate individual variance and reduce aleatoric errors. Furthermore, by meticulously modeling relation-level interactions across modalities, we construct robust cross-modal features that significantly enhance recognition performance. We evaluate our proposed SUMRR framework on notable cross-modal, egocentric sports datasets using various backbone architectures and achieve a remarkable 89.49% recognition precision. Experimental results demonstrate the portability and robustness of our SUMRR for egocentric human recognition in sports.