A mixed reality-based aircraft cable harness installation assistance system with fully occluded gesture recognition

Abstract

In limited visibility human-machine environments, there has been little discussion on hand motion parameter extraction, behavioral intention data analysis, and the effectiveness of 3D assembly instructions. To address this issue, we developed a mixed reality system for fully occluded gesture recognition (Fog-MR), which supports the extraction of complete hand motion models, the determination of a directional relationship between hand motion models and task intentions, and provides timely, natural visual feedback for hand operations. Firstly, a 3D hand pose registration method for cyber-physical objects is proposed, which uses BundleFusion to obtain a 3D point cloud model of virtual hands in VR space. Secondly, a hand motion feature spectral clustering analysis method based on a hand reconstruction model is constructed, combining a dual autoencoder network and mutual information metrics to achieve precise matching of hand behavioral intentions. Finally, a new industrial mixed reality visual prompt is designed, providing operators with more vivid and specific operational guidance. Experimental data indicate that compared to our previously developed mixed reality assembly system (Ug-MR), Fog-MR system significantly improves the motion synchronization accuracy between virtual hands and operator's hands, the accuracy of clustering hand motion parameters, and the naturalness of 3D assembly instructions that express behavioral intentions. These improvements significantly reduce the frequency of errors and omissions in such human-machine collaborative assembly processes, which is crucial for enhancing the reliability of high-precision manual operations.