A Global–Local Fusion Model Exploring Temporal–Spatial Dependence for Multimodal Hand Gesture Recognition

Hand Gesture Recognition (HGR) employing surface electromyography (sEMG) and accelerometer (ACC) signals has garnered increasing interest in areas of bionic prostheses and human-machine interaction. However, existing multimodal approaches predominantly extract global specificity at a single temporal scale, which neglects local dynamic characteristics. This limitation hinders the effective capture of global-local temporal information, resulting in restricted performance and frequent misclassification of dynamic gestures. To this end, we propose a novel global-local Fusion model, termed Temporal-spatial Dependence Fusion (TsdFusion), for sEMG-ACC-based HGR. TsdFusion harnesses temporal-spatial dependencies (Tsd) from multi-time scale handcrafted features and employs a Convolution-Transformer framework for global-local fusion, thus enriching local dynamic information while preserving global insights. Specifically, the Tsd inputs are independently constructed from sEMG and ACC through multi-time scale window segmentation and feature engineering. Furthermore, the global and local temporal-spatial correlations within unimodal Tsd inputs are characterized by the unimodal transformer and dimension-wise convolution modules, respectively. Subsequently, a Convolution-coupled-transformer progressive hierarchical fusion module effectively integrates intramodal specificity and intermodal hierarchical relationship for final prediction. Evaluations on four public datasets, including transradial amputees and healthy subjects, demonstrate TsdFusion outperforms the state-of-the-art multimodal HGR methods. The TsdFusion effectively recognizes dynamic gestures, facilitating promising HGR-based interaction for prostheses or assistance robotics.