A mixed reality-based aircraft cable harness installation assistance system with fully occluded gesture recognition

IF 9.1 1区 计算机科学 Q1 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Zhuo Wang , Weichu Li , Jiacheng Zhang , Yiliang Zhou , Shisong Chen , Yuwei Dai , Jiale Song , Yeming Cheng , Xiaoting Du
{"title":"A mixed reality-based aircraft cable harness installation assistance system with fully occluded gesture recognition","authors":"Zhuo Wang ,&nbsp;Weichu Li ,&nbsp;Jiacheng Zhang ,&nbsp;Yiliang Zhou ,&nbsp;Shisong Chen ,&nbsp;Yuwei Dai ,&nbsp;Jiale Song ,&nbsp;Yeming Cheng ,&nbsp;Xiaoting Du","doi":"10.1016/j.rcim.2024.102930","DOIUrl":null,"url":null,"abstract":"<div><div>In limited visibility human-machine environments, there has been little discussion on hand motion parameter extraction, behavioral intention data analysis, and the effectiveness of 3D assembly instructions. To address this issue, we developed a mixed reality system for fully occluded gesture recognition (Fog-MR), which supports the extraction of complete hand motion models, the determination of a directional relationship between hand motion models and task intentions, and provides timely, natural visual feedback for hand operations. Firstly, a 3D hand pose registration method for cyber-physical objects is proposed, which uses BundleFusion to obtain a 3D point cloud model of virtual hands in VR space. Secondly, a hand motion feature spectral clustering analysis method based on a hand reconstruction model is constructed, combining a dual autoencoder network and mutual information metrics to achieve precise matching of hand behavioral intentions. Finally, a new industrial mixed reality visual prompt is designed, providing operators with more vivid and specific operational guidance. Experimental data indicate that compared to our previously developed mixed reality assembly system (Ug-MR), Fog-MR system significantly improves the motion synchronization accuracy between virtual hands and operator's hands, the accuracy of clustering hand motion parameters, and the naturalness of 3D assembly instructions that express behavioral intentions. These improvements significantly reduce the frequency of errors and omissions in such human-machine collaborative assembly processes, which is crucial for enhancing the reliability of high-precision manual operations.</div></div>","PeriodicalId":21452,"journal":{"name":"Robotics and Computer-integrated Manufacturing","volume":"93 ","pages":"Article 102930"},"PeriodicalIF":9.1000,"publicationDate":"2024-12-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Robotics and Computer-integrated Manufacturing","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0736584524002175","RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

In limited visibility human-machine environments, there has been little discussion on hand motion parameter extraction, behavioral intention data analysis, and the effectiveness of 3D assembly instructions. To address this issue, we developed a mixed reality system for fully occluded gesture recognition (Fog-MR), which supports the extraction of complete hand motion models, the determination of a directional relationship between hand motion models and task intentions, and provides timely, natural visual feedback for hand operations. Firstly, a 3D hand pose registration method for cyber-physical objects is proposed, which uses BundleFusion to obtain a 3D point cloud model of virtual hands in VR space. Secondly, a hand motion feature spectral clustering analysis method based on a hand reconstruction model is constructed, combining a dual autoencoder network and mutual information metrics to achieve precise matching of hand behavioral intentions. Finally, a new industrial mixed reality visual prompt is designed, providing operators with more vivid and specific operational guidance. Experimental data indicate that compared to our previously developed mixed reality assembly system (Ug-MR), Fog-MR system significantly improves the motion synchronization accuracy between virtual hands and operator's hands, the accuracy of clustering hand motion parameters, and the naturalness of 3D assembly instructions that express behavioral intentions. These improvements significantly reduce the frequency of errors and omissions in such human-machine collaborative assembly processes, which is crucial for enhancing the reliability of high-precision manual operations.
基于混合现实技术的飞机电缆线束安装辅助系统,具有完全隐蔽的手势识别功能
在有限可见的人机环境中,对手部运动参数提取、行为意向数据分析以及三维装配指令的有效性的讨论很少。为了解决这个问题,我们开发了一个用于全遮挡手势识别(Fog-MR)的混合现实系统,该系统支持完整手部运动模型的提取,手部运动模型与任务意图之间的方向关系的确定,并为手部操作提供及时、自然的视觉反馈。首先,提出了一种面向网络物理对象的三维手姿配准方法,利用BundleFusion技术获得虚拟手在VR空间中的三维点云模型;其次,构建了基于手部重构模型的手部运动特征谱聚类分析方法,结合双自编码器网络和互信息度量实现手部行为意图的精确匹配;最后,设计了一种新型的工业混合现实视觉提示,为操作者提供更加生动、具体的操作指导。实验数据表明,与我们之前开发的混合现实装配系统(Ug-MR)相比,Fog-MR系统显著提高了虚拟手与操作者手之间的运动同步精度、手部运动参数聚类的精度以及表达行为意图的3D装配指令的自然性。这些改进大大减少了这种人机协作装配过程中错误和遗漏的频率,这对于提高高精度手动操作的可靠性至关重要。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Robotics and Computer-integrated Manufacturing
Robotics and Computer-integrated Manufacturing 工程技术-工程:制造
CiteScore
24.10
自引率
13.50%
发文量
160
审稿时长
50 days
期刊介绍: The journal, Robotics and Computer-Integrated Manufacturing, focuses on sharing research applications that contribute to the development of new or enhanced robotics, manufacturing technologies, and innovative manufacturing strategies that are relevant to industry. Papers that combine theory and experimental validation are preferred, while review papers on current robotics and manufacturing issues are also considered. However, papers on traditional machining processes, modeling and simulation, supply chain management, and resource optimization are generally not within the scope of the journal, as there are more appropriate journals for these topics. Similarly, papers that are overly theoretical or mathematical will be directed to other suitable journals. The journal welcomes original papers in areas such as industrial robotics, human-robot collaboration in manufacturing, cloud-based manufacturing, cyber-physical production systems, big data analytics in manufacturing, smart mechatronics, machine learning, adaptive and sustainable manufacturing, and other fields involving unique manufacturing technologies.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信