Self-Supervised Learning for Intuitive Control of Prosthetic Hand Movements via Sonomyography

Abstract

As a primary effector of humans, the hand plays a crucial role in many aspects of daily life. Recognizing multidegree-of-freedom hand movements from muscle activity helps infer human motion intentions. Solving this problem has direct applications in prosthetic and exoskeleton control. Here, we propose a self-supervised learning algorithm inspired by muscle synergies to achieve simultaneous estimation of wrist rotation (supination/pronation) and hand grasp (open/close) from sonomyography—the muscle deformation detected by a wearable ultrasound array. Unlike conventional methods collecting both muscle activity and hand kinematics for supervised model calibration, this algorithm only uses unlabeled forearm ultrasound signals for self-supervised wrist and hand movement estimation, where movement labels are auto-generated. The performance of the proposed algorithm was experimentally evaluated with ten participants including an amputee. Offline analysis demonstrated that the proposed algorithm can accurately estimate simultaneous wrist rotation and hand grasp movements ( $r_{\textrm {wrist}}$ and $r_{\textrm {hand}}$ were 0.98 and 0.94 for the able-bodied, and 0.98 and 0.90 for the amputee, respectively). Notably, the performance of the self-supervised learning was superior to the supervised learning for the amputee. Online experiments demonstrated that intended wrist and hand movements can be deciphered in real time, enabling accurate control of a virtual hand. This study will open up a new avenue for the sonomyographic human-machine interaction.