Temporally enhanced graph convolutional network for hand tracking from an egocentric camera

IF 4.4 3区 计算机科学 Q1 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS
Woojin Cho, Taewook Ha, Ikbeom Jeon, Jinwoo Jeon, Tae-Kyun Kim, Woontack Woo
{"title":"Temporally enhanced graph convolutional network for hand tracking from an egocentric camera","authors":"Woojin Cho, Taewook Ha, Ikbeom Jeon, Jinwoo Jeon, Tae-Kyun Kim, Woontack Woo","doi":"10.1007/s10055-024-01039-3","DOIUrl":null,"url":null,"abstract":"<p>We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2.</p>","PeriodicalId":23727,"journal":{"name":"Virtual Reality","volume":null,"pages":null},"PeriodicalIF":4.4000,"publicationDate":"2024-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Virtual Reality","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1007/s10055-024-01039-3","RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}
引用次数: 0

Abstract

We propose a robust 3D hand tracking system in various hand action environments, including hand-object interaction, which utilizes a single color image and a previous pose prediction as input. We observe that existing methods deterministically exploit temporal information in motion space, failing to address realistic diverse hand motions. Also, prior methods paid less attention to efficiency as well as robust performance, i.e., the balance issues between time and accuracy. The Temporally Enhanced Graph Convolutional Network (TE-GCN) utilizes a 2-stage framework to encode temporal information adaptively. The system establishes balance by adopting an adaptive GCN, which effectively learns the spatial dependency between hand mesh vertices. Furthermore, the system leverages the previous prediction by estimating the relevance across image features through the attention mechanism. The proposed method achieves state-of-the-art balanced performance on challenging benchmarks and demonstrates robust results on various hand motions in real scenes. Moreover, the hand tracking system is integrated into a recent HMD with an off-loading framework, achieving a real-time framerate while maintaining high performance. Our study improves the usability of a high-performance hand-tracking method, which can be generalized to other algorithms and contributes to the usage of HMD in everyday life. Our code with the HMD project will be available at https://github.com/UVR-WJCHO/TEGCN_on_Hololens2.

Abstract Image

时态增强图卷积网络,用于从自我中心摄像机追踪手部图像
我们提出了一种在各种手部动作环境(包括手与物体的交互)中使用的鲁棒三维手部跟踪系统,该系统利用单张彩色图像和之前的姿势预测作为输入。我们发现,现有的方法只能确定性地利用运动空间中的时间信息,无法解决现实中多种多样的手部动作。此外,以前的方法不太注重效率和稳健性能,即时间和准确性之间的平衡问题。时序增强图卷积网络(TE-GCN)采用两阶段框架对时序信息进行自适应编码。该系统通过采用自适应 GCN 来建立平衡,从而有效地学习手部网格顶点之间的空间依赖性。此外,该系统还通过注意力机制估算图像特征之间的相关性,从而利用先前的预测。所提出的方法在具有挑战性的基准测试中实现了最先进的平衡性能,并在真实场景中的各种手部运动中展示了稳健的结果。此外,手部跟踪系统通过卸载框架集成到最新的 HMD 中,在保持高性能的同时实现了实时帧速率。我们的研究提高了高性能手部跟踪方法的可用性,该方法可推广到其他算法,并有助于 HMD 在日常生活中的使用。我们与 HMD 项目的代码将发布在 https://github.com/UVR-WJCHO/TEGCN_on_Hololens2 上。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Virtual Reality
Virtual Reality COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS-COMPUTER SCIENCE, SOFTWARE ENGINEERING
CiteScore
8.30
自引率
14.30%
发文量
95
审稿时长
>12 weeks
期刊介绍: The journal, established in 1995, publishes original research in Virtual Reality, Augmented and Mixed Reality that shapes and informs the community. The multidisciplinary nature of the field means that submissions are welcomed on a wide range of topics including, but not limited to: Original research studies of Virtual Reality, Augmented Reality, Mixed Reality and real-time visualization applications Development and evaluation of systems, tools, techniques and software that advance the field, including: Display technologies, including Head Mounted Displays, simulators and immersive displays Haptic technologies, including novel devices, interaction and rendering Interaction management, including gesture control, eye gaze, biosensors and wearables Tracking technologies VR/AR/MR in medicine, including training, surgical simulation, rehabilitation, and tissue/organ modelling. Impactful and original applications and studies of VR/AR/MR’s utility in areas such as manufacturing, business, telecommunications, arts, education, design, entertainment and defence Research demonstrating new techniques and approaches to designing, building and evaluating virtual and augmented reality systems Original research studies assessing the social, ethical, data or legal aspects of VR/AR/MR.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信