IoT-Integrated Augmented Reality for Spatially Aware Human–Machine Interaction for IEEE Transactions on Human-Machine Systems

Abstract

A human–machine interface (HMI) is an essential component of any modern industrial machine, acting as the portal for human workers to setup, monitor, control, and troubleshoot machines. Yet, existing HMIs often pose challenges in usability and learnability due to overwhelming data, nonstandard interfaces, and physical design limitations. We argue that these challenges stem from a lack of spatiotemporal alignment of data and controls with the machine, leading to information overload and loss as the operator repeatedly translates signals between the HMI and machine. We propose a novel augmented reality (AR) interface integrated with Internet of Things (IoT), which affords spatially aware, natural mapping of real-time data and controls with the machine while providing visibility and immediate feedback. We build the IoT-integrated AR system through a Kepware-ThingWorx-Vuforia pipeline implemented in a model cyber-physical factory for cellphone assembly. We conducted a between-subjects user study with 20 participants split into two groups, who operated three assembly line stations using our system deployed on HoloLens 2 and the machine's built-in Siemens HMIs in opposite orders. The findings show the significant effects of our AR interfaces on improving efficiency, lowering task completion times, reducing errors, and enhancing usability and mental load. We conclude the paper by discussing the limitations and several directions for future research on IoT-integrate AR interfaces to transform how humans interact with industrial machines.