Dynamic gesture recognition during human–robot interaction in autonomous earthmoving machinery used for construction

Abstract

Effective interaction between operators and autonomous earthmoving machinery can accurately convey the rich engineering experience of operators to machines, ensuring efficient human–robot collaboration in construction. In this study, we propose a pipeline for dynamic gesture interaction between authorised operators and autonomous earthmoving machinery. Initially, the autonomous earthmoving machinery preprocessed the video stream using video restoration algorithms if it operated under harsh environmental conditions. Subsequently, the machinery used a safety helmet colour detection algorithm based on YOLOv8 to determine whether an operator has the authorisation to interact with it by recognising the colour of the safety helmet worn by the operator, thereby preventing incorrect operations of the machinery from unauthorised operators. Finally, the autonomous earthmoving machinery utilised the proposed video swin transformer with Adapt multilayer perceptron (AdaptViSwT) dynamic gesture recognition algorithm to recognise dynamic gesture instructions provided by authorised operators and execute the corresponding operations, enabling human–robot collaboration under complex construction conditions. To train the proposed AdaptViSwT effectively, we established a dynamic gesture interaction dataset comprising 6,502 videos that contained nine commonly used instructions for commanding earthmoving machinery. The experiments verified that, on construction-site datasets, the proposed pipeline achieved 91.2% accuracy in detecting authorised worker. In dynamic gesture recognition, it achieved 98.32% accuracy and 98.44% F1-score. These results effectively ensure the safety and reliability of human-robot collaborative construction.