Human-Robot Interaction: Integrating BCI-Controlled Virtual Robotic Arm With Manual Control in VR Environment

Abstract

This study presents a novel tri-manual interaction framework that enables users to control two physical hands via VR controllers and a virtual robotic arm through a hybrid brain-computer interface (32-channel EEG and eye-tracking). The virtual robotic arm, implemented as a 6-DOF industrial manipulator in Unity, is controlled through simplified BCI commands: forward/backward movement along the z-axis based on motor imagery strength, with automatic grasping triggered by sustained attention thresholds. Twenty-five participants completed 30 trials, each following a 60-s protocol with five phases: rest, target presentation, preparation, execution, and feedback. Results demonstrated that the hybrid BCI system achieved superior performance compared to EEG-only control: 8.5% improvement in task success rate, 34.5% increase in positioning accuracy, and 46.2% reduction in cognitive load. The CNN-LSTM architecture achieved 86.3% motor imagery classification accuracy. Learning effects were observed within trials, with performance plateauing after 3.8 ± 0.7 attempts. Time-frequency analysis revealed hierarchical neural coordination mechanisms underlying the tri-manual control. This research validates the feasibility of augmented manipulation in virtual environments, establishing a foundation for advanced human-robot collaboration in animation and VR applications.