Probing AR-assisted seamless HRC assembly for industry 5.0: Multi-modal mutual cognition and LLM-driven knowledge reasoning

The advancement of Industry 5.0 fosters human-centric manufacturing, aiming to enhance the well-being and needs of operators. Current research on Human-robot Collaboration (HRC) in the assembly scene is progressively evolving toward intelligent manufacturing with a human-centric focus, particularly in safety and interaction. However, in existing HRC assembly environments, the disjointed relationship between humans and robots presents challenges in handling complex manufacturing tasks. Cobots often struggle to accurately comprehend human actions and assembly contexts, while operators lack real-time insights into the current assembly scene. This mismatch increases the cognitive and physical burden on operators, ultimately reducing assembly efficiency and quality. To address this issue and achieve seamless HRC assembly, this paper proposes an Augmented Reality (AR) assisted HRC assembly method that integrates multimodal mutual cognition with Large Language Model (LLM) driven knowledge reasoning. Firstly, a Transformer-based multimodal fusion perception method for HRC scenarios is proposed to overcome the limitations of current cobots in understanding and responding to human commands and environmental changes in real-time. Based on this, a human-centric mutual cognition and safety framework for HRC is proposed to mitigate interaction risks and promote intelligent coexistence between operators and cobots. Secondly, an LLM-driven assembly knowledge reasoning system is proposed. By constructing an assembly craft information model with semantic associations and dynamic updating capabilities, the system facilitates decision support for HRC tasks and provides optimized assembly plan recommendations. This approach effectively leverages the respective advantages of human operators and cobots in executing assembly tasks. In addition, an AR-based HRC assembly assistance system is designed to enhance visual guidance during the assembly process while integrating the functionalities above. Finally, practical validation is conducted in real-world assembly tasks. Experimental results demonstrate that the proposed method offers efficient, intelligent, and visualized support for HRC assembly tasks, significantly reducing the cognitive load on operators while improving assembly efficiency and product consistency.