Parallel collaboration and closed-loop control of a cursor using multimodal physiological signals

Abstract

This paper explores the parallel collaboration of multimodal physiological signals, combining eye tracker output signals, motor imagery, and error-related potentials to control a computer mouse. Specifically, a parallel working mechanism is implemented in the decision layer, where the eye tracker manages cursor movements, and motor imagery manages click functions. Meanwhile, the eye tracker output signals are integrated with electroencephalography data to detect the idle state for asynchronous control. Additionally, error-related potentials evoked by visual feedback, are detected to reduce the cost of error corrections. To efficiently collect data and provide continuous evaluations, we performed offline training and online testing in the designed paradigm. To further validate the practicability, we conducted online experiments on the real-world computer, focusing on a scenario of opening and closing files. The experiments involved seventeen subjects. The results showed that the stability of the eye tracker was optimized from 67.6% to 95.2% by the designed filter, providing the support for parallel control. The accuracy of motor imagery conducted simultaneously with fixations reached 93.41 ± 2.91%, proving the feasibility of parallel control. Furthermore, the real-world experiments took 45.86 ± 14.94 s to complete three movements and clicks, and showed a significant improvement compared to the baseline experiment without automatic error correction, validating the practicability of the system and the efficacy of error-related potentials detection. Moreover, this system freed users from the stimulus paradigm, enabling a more natural interaction. To sum up, the parallel collaboration of multimodal physiological signals is novel and feasible, the designed mouse is practical and promising.