Multimodal Deep Learning Model for Cylindrical Grasp Prediction Using Surface Electromyography and Contextual Data During Reaching.

Abstract

Grasping objects, from simple tasks to complex fine motor skills, is a key component of our daily activities. Our approach to facilitate the development of advanced prosthetics, robotic hands and human-machine interaction systems consists of collecting and combining surface electromyography (EMG) signals and contextual data of individuals performing manipulation tasks. In this context, the identification of patterns and prediction of hand grasp types is crucial, with cylindrical grasp being one of the most common and functional. Traditional approaches to grasp prediction often rely on unimodal data sources, limiting their ability to capture the complexity of real-world scenarios. In this work, grasp prediction models that integrate both EMG signals and contextual (task- and product-related) information have been explored to improve the prediction of cylindrical grasps during reaching movements. Three model architectures are presented: an EMG processing model based on convolutions that analyzes forearm surface EMG data, a fully connected model for processing contextual information, and a hybrid architecture combining both inputs resulting in a multimodal model. The results show that context has great predictive power. Variables such as object size and weight (product-related) were found to have a greater impact on model performance than task height (task-related). Combining EMG and product context yielded better results than using each data mode separately, confirming the importance of product context in improving EMG-based models of grasping.