RGB video and inertial sensing fusion method for human action recognition in human-robot collaborative manufacturing

Abstract

Human action recognition (HAR), as a prerequisite for robotic dynamic decision-making, is crucial for achieving efficient human-robot collaborative manufacturing (HRCM). Compared with single modality, multi-modality provides a more comprehensive understanding of human actions. However, it is a challenge to effectively integrate this information to fully leverage the advantages of multi-modality for HAR in HRCM. Therefore, in this paper, the RGB video and inertial sensing fusion method for HAR in HRCM is proposed, presenting the systematic exploration of this multi-modality in industrial contexts. Two fusion strategies of two modalities are studied: decision-level fusion and feature-level fusion. Secondly, taking the rotary vector (RV) reducer assembly as an example, a multi-modal human assembly action dataset for HAR (HAAD-SDU) is designed, filling the gap in the HRCM field where publicly representative datasets are scarce. This dataset synchronously introduces RGB video and inertial sensing data containing human assembly information. Finally, the feasibility and effectiveness of the proposed approach are verified by the designed dataset and public dataset, demonstrating superior performance over baseline methods. The experimental results demonstrate that the proposed fusion approach integrating RGB video and inertial sensing modalities not only overcomes the limitations of the single modality but also exhibits strong cross-domain generalizability, proving effective for both industrial tasks and daily activity recognition. In the HRCM scenario specifically, both decision-level and feature-level fusion strategies demonstrate superior recognition capabilities. The decision-level fusion provides a higher recognition accuracy of 95.71 %, while the feature-level fusion achieves competitive accuracy at 94.42 % with low recognition latency of 1.67 s. Notably, the proposed fusion model can accurately recognize human behaviors at least 2 s before they are completed, providing sufficient leftover time for the robotic system to complete collaborative tasks.