Copilot: A framework for integrating LLM and BMI to enhance human–robot interaction

Abstract

This paper proposes an innovative human–robot interaction (HRI) framework called Copilot, which aims to bridge the gap between human intent and robot intelligence. Currently, existing HRI systems struggle to infer human intentions and rely heavily on predefined rules, a limitation that significantly hinders the advancement of the field. To address this issue, the Copilot framework, for the first time, integrates the environmental understanding capabilities of large language models (LLMs) with the intention recognition advantages of brain-machine interface (BMI). It constructs three core modules: (1) a LLM-based visual evoked potential (LLM-VEP) paradigm module utilizing LLM for scene understanding and dynamic marking; (2) a BMI module employing the blink-triggered multivariate variational mode decomposition with canonical correlation analysis (BT-MVMD-CCA) algorithm; and (3) an intelligent agent flexibly adapting to different task requirements. Through online experimental validation with 12 participants, the system performed optimally when using the EEG-based double blink triggering (EEG-DBT) method: 0% false trigger rate, 94.09% blink detection rate, and 84.00% task completion rate. In offline experiments, the proposed BT-MVMD-CCA algorithm achieved 92.3% classification accuracy and a peak information transfer rate (ITR) of 71.1 bits/min at DTW = 1.5 s. This research not only provides theoretical support for the HRI field, but also offers promising solutions for assistive robotics and manufacturing scenarios.