An immersive spatially consistent multi-modal augmented virtuality human-machine interface for telerobotic systems

Abstract

This study presents an immersive augmented virtuality (AV)-based human-machine interface (HMI) designed to enhance telepresence and operator performance in telerobotic systems. Traditional telerobotic systems often face limitations such as 2D representations of 3D environments, restricted fields of view, and reduced depth perception, all of which hinder operator effectiveness. Although extended reality and various augmentation technologies have been employed to create more intuitive teleoperations, prior research has largely overlooked the integration of spatially consistent video streams from remote sites, which significantly increases operators' mental workload. As a result, these systems struggled to manage dynamic changes at the remote site and lacked sufficient environmental context and an unlimited field of view for operators. This study addresses these limitations by augmenting the virtual replica of the remote environment with a real-time, spatially consistent video stream within the AV-based HMI, enabling operators to better understand dynamic changes at the remote site and enhancing both situational awareness and control precision during teleoperations. Additionally, 3D point clouds and haptic feedback are integrated to create a multi-modal interface that further improves operator perception and interaction with the remote environment. A user study comparing the immersive AV-based HMI with a multi-monocular HMI demonstrated significant improvements in task workload, system usability, spatial presence, and task completion times. Participant feedback further confirmed the system’s ability to improve operator performance.