On the Benefit of FMG and EMG Sensor Fusion for Gesture Recognition Using Cross-Subject Validation

Abstract

Hand gestures are a natural form of human communication, making gesture recognition a sensible approach for intuitive human-computer interaction. Wearable sensors on the forearm can be used to detect the muscle contractions that generate these gestures, but classification approaches relying on a single measured modality lack accuracy and robustness. In this work, we analyze sensor fusion of force myography (FMG) and electromyography (EMG) for gesture recognition. We employ piezoelectric FMG sensors based on ferroelectrets and a commercial EMG system in a user study with 13 participants to measure 66 distinct hand movements with 10ms labelling precision. Three classification tasks, namely flexion and extension, single finger, and all finger movement classification, are performed using common handcrafted features as input to machine learning classifiers. Subsequently, the evaluation covers the effectiveness of the sensor fusion using correlation analysis, classification performance based on leave-one-subject-out-cross-validation and 5x2cv-t-tests, and its effects of involuntary movements on classification. We find that sensor fusion leads to significant improvement (42% higher average recognition accuracy) on all three tasks and that both sensor modalities contain complementary information. Furthermore, we confirm this finding using reduced FMG and EMG sensor sets. This study reinforces the results of prior research about the effectiveness of sensor fusion by performing meticulous statistical analyses, thereby paving the way for multi-sensor gesture recognition in assistance systems.