Toward Complex In-Car Environment Human–Vehicle Interactions Through Smart Glasses and sEMG-Based Gesture Recognition

Abstract

Visual distraction is a significant cause of traffic accidents, which draws extensive research dedicated to devising a safe and efficient means of human–vehicle interaction. To achieve robust gesture interaction adaptability in complex automotive environments, this article proposes a novel weighted random forest (WRF) multimodal fusion method, which integrates data from smart glasses and surface electromyography (sEMG) to recognize gestures in various in-vehicle settings. Image and sEMG data of various gestures are processed and extracted with a parallel feature extraction module to compute the recognition probabilities under different scenarios, and then, the WRF is utilized for the final classification result. The experimental results indicate that the classification accuracy of the fusion method reaches as high as 96.48% in normal, occluded, and low-light (nighttime) scenarios. Furthermore, the method proposed in this article exhibits superior performance among all comparative fusion strategies. Compared with the baseline method, its recognition accuracy is improved by up to 3.35%. This provides feasibility for robust human–machine gesture interaction in various limited scenarios and paves the way for future full-scene deployment of human–machine interaction.