ProjecTwin: A digital twin-based projection framework for flexible spatial augmented reality in adaptive assistance

Abstract

Establishing an efficient communication channel between the digital manufacturing system and assembly workers is a key research area in smart assembly. Spatial augmented reality (SAR) technology offers a naked-eye display mode that seamlessly integrates virtual information into real industrial scenes, providing an immersive experience for multiple viewers. However, the challenge arises when facing frequent workstation reconfiguration in customized product assembly tasks, requiring a flexible spatial augmented reality (FSAR) mode. To address the issues of uncertainty in projection layout, distortion in projected display, and lack of mobility in the projector of FSAR, a DT (digital twin)-based projection framework called ProjecTwin is proposed. The ProjecTwin framework can simulate real-time projection scenarios and enables the reconfiguration of projection layouts. Based on ProjecTwin, an FSAR assistance method is proposed. An optimization model is employed to solve the DT projection layout issue by considering human-robot collaboration scenarios as constraints. Additionally, a method for reverse generation of DT images is proposed to address the image distortion problem. Furthermore, a DT entity control method based on mobile robots is introduced to tackle the inflexibility of the projector. Three cases are conducted to illustrate how the proposed method supports optimized, adaptive, and human-centric FSAR assistance.