A human-robot collaborative assembly framework with quality checking based on real-time dual-hand action segmentation

Abstract

This paper presents a human-robot collaborative assembly (HRCA) framework, addressing key challenges in real-time dual-hand action understanding, adaptive robot assistance, and in-process quality checking. At its core is DuHa-v2, a real-time dual-hand action segmentation algorithm that efficiently segments assembly actions of two hands by integrating object interaction and action features. DuHa-v2 enables robots to proactively assist human workers by utilising either next-task prediction or indicative action recognition, both informed by the segmented action sequences. An in-process quality checking mechanism is proposed to ensure high assembly quality and efficiency by identifying errors immediately after critical assembly steps. The framework's effectiveness is validated through experiments on both the HA-ViD dataset and a real-world case study, demonstrating superior dual-hand action segmentation performance, timely robot assistance, and effective quality checking. The proposed HRCA framework enables robots to collaborate with humans in a more intuitive and reliable way by providing timely assistance, whether or not the overall task is known, and performing in-time assembly quality checks. More information can be found in https://github.com/hao-zheng-research/A-human-robot-collaborative-assembly-framework-with-quality-checking.