Developing Brain-Based Bare-Handed Human-Machine Interaction via On-Skin Input

Abstract

Developing natural, intuitive, and human-centric input systems for mobile human-machine interaction (HMI) poses significant challenges. Existing gaze or gesture-based interaction systems are often constrained by their dependence on continuous visual engagement, limited interaction surfaces, or cumbersome hardware. To address these challenges, we propose MetaSkin, a novel neurohaptic interface that uniquely integrates neural signals with on-skin interaction for bare-handed, eyes-free interaction by exploiting human’s natural proprioceptive capabilities. To support the interface, we developed a deep learning framework that employs multiscale temporal-spectral feature representation and selective feature attention to effectively decode neural signals generated by on-skin touch and motion gestures. In experiments with 12 participants, our method achieved offline accuracies of 81.95% for touch location discrimination, 71.00% for motion type identification, and 46.08% for 10-class touch-motion classification. In pseudo-online settings, accuracies reached 99.43% for touch onset detection, and 80.34% and 67.02% for classification of touch location and motion type, respectively. Neurophysiological analyses revealed distinct neural activation patterns in the sensorimotor cortex, underscoring the efficacy of our multiscale approach in capturing rich temporal and spectral dynamics. Future work will focus on optimizing the system for diverse user populations and dynamic environments, with a long-term goal of advancing human-centered, neuroadaptive interfaces for next-generation HMI systems. This work represents a significant step toward a paradigm shift in design of brain-computer interfaces, bridging sensory and motor paradigms for building more sophisticated systems.