A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction

Abstract

Due to the complexity of the task and the diversification of the scene, the traditional interactive control method can no longer meet the requirements of users. To solve this problem, a multi-modal virtual-real fusion system for multi-task human-computer interaction is proposed in this paper, which integrates eye movement, gesture and voice. Frist, aiming at the phenomenon of multi-task and multi-modal, a task-modal matching model is established. Then, the task-modal matching model is abstracted into a multi-objective optimization problem, and a method for solving this problem is designed and a matching scheme is successfully obtained. Meanwhile, the construction of the system is completed for the virtual-real fusion environment, and the control of unmanned car and virtual car is realized. The system can carry out multi-modal interaction and complete multiple tasks in real scene, virtual scene, parallel system and virtual-real fusion scene. Finally, the experiment proves the stability and reliability of the system.