Enhancing industrial human action recognition framework integrating skeleton data acquisition, data repair and optimized graph convolutional networks

Abstract

The precise interpretation of human actions is crucial for seamless interaction and operational efficiency for industrial human-robot collaboration. However, existing skeleton-based action recognition methods focus on algorithmic applications while overlooking key challenges such as robust data acquisition, validation, and repair. Additionally, the scarcity of high-quality industrial datasets and the challenges in distinguishing similar actions further limit the capability to infer operators' intentions accurately. This paper presents a novel framework to address challenges utilizing integrating skeleton data acquisition, effective data augmentation method, and an optimized graph convolutional network. Specifically, the proposed framework employs a pose estimation method for 2D (two-dimensional) joint estimation and a 2D-to-3D (three-dimensional) lifting technique, supplemented with a robust method for repairing invalid skeleton data and a skeletal feature-based data augmentation strategy. To enhance action recognition, this paper introduces the Channel-Topology Refinement Graph Convolutional Network Plus (CTR-GCN-Plus), which incorporates dynamic topology learning and multi-channel feature aggregation, augmented with hand motion integration for finer differentiation of similar actions. The proposed framework is evaluated on an industrial assembly dataset incorporating challenging scenarios, such as occlusions and similar actions. Experimental results demonstrate that the proposed methods significantly improve accuracy, enhance recognition of similar actions, and effectively account for individual variations, outperforming existing approaches in industrial human-robot collaboration environments.