A Vision-based Human Digital Twin Modelling Approach for Adaptive Human-Robot Collaboration

Human-robot collaboration (HRC) has been regarded as one of the most promising paradigms for human-centric smart manufacturing in the context of Industry 5.0. To improve human well-being and robotic flexibility in HRC, a plethora of works around human body perception have emerged over the years, but most of them only considered a specific facade of human recognition while lacking a holistic perspective of the human operator. To this end, this study proposes an exemplary vision-based Human Digital Twin (HDT) model for highly dynamic HRC applications. The model mainly consists of a convolutional neural network that can simultaneously model the hierarchical human status including 3D human posture, action intention, and ergonomic risk. Then, on the basis of the constructed HDT, a robotic motion planning strategy is further introduced with the aim of adaptively optimizing the robotic motion trajectory. Further experiments and case studies are conducted in an HRC scenario to demonstrate the effectiveness of our approach.