2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)最新文献_第3页

STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising STGAE:用于手部运动去噪的时空图自编码器

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00018

Kanglei Zhou, Zhiyuan Cheng, Hubert P. H. Shum, Frederick W. B. Li, Xiaohui Liang

{"title":"STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising","authors":"Kanglei Zhou, Zhiyuan Cheng, Hubert P. H. Shum, Frederick W. B. Li, Xiaohui Liang","doi":"10.1109/ismar52148.2021.00018","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00018","url":null,"abstract":"Hand object interaction in mixed reality (MR) relies on the accurate tracking and estimation of human hands, which provide users with a sense of immersion. However, raw captured hand motion data always contains errors such as joints occlusion, dislocation, high-frequency noise, and involuntary jitter. Denoising and obtaining the hand motion data consistent with the user’s intention are of the utmost importance to enhance the interactive experience in MR. To this end, we propose an end-to-end method for hand motion denoising using the spatial-temporal graph auto-encoder (STGAE). The spatial and temporal patterns are recognized simultaneously by constructing the consecutive hand joint sequence as a spatial-temporal graph. Considering the complexity of the articulated hand structure, a simple yet effective partition strategy is proposed to model the physic-connected and symmetry-connected relationships. Graph convolution is applied to extract structural constraints of the hand, and a self-attention mechanism is to adjust the graph topology dynamically. Combining graph convolution and temporal convolution, a fundamental graph encoder or decoder block is proposed. We finally establish the hourglass residual auto-encoder to learn a manifold projection operation and a corresponding inverse projection through stacking these blocks. In this work, the proposed framework has been successfully used in hand motion data denoising with preserving structural constraints between joints. Extensive quantitative and qualitative experiments show that the proposed method has achieved better performance than the state-of-the-art approaches.","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123926117","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Blending Shadows: Casting Shadows in Virtual and Real using Occlusion-Capable Augmented Reality Near-Eye Displays 混合阴影:使用具有遮挡能力的增强现实近眼显示器在虚拟和真实中投射阴影

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00059

Kiyosato Someya, Yuta Itoh

引用次数: 1

A Predictive Performance Model for Immersive Interactions in Mixed Reality 混合现实中沉浸式交互的预测性能模型

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00035

Florent Cabric, E. Dubois, M. Serrano

{"title":"A Predictive Performance Model for Immersive Interactions in Mixed Reality","authors":"Florent Cabric, E. Dubois, M. Serrano","doi":"10.1109/ismar52148.2021.00035","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00035","url":null,"abstract":"The design of immersive interaction for mixed reality based on head-mounted displays (HMDs), hereafter referred to as Mixed Reality (MR), is still a tedious task which can hinder the advent of such devices. Indeed, the effects of the interface design on task performance are difficult to anticipate during the design phase: the spatial layout of virtual objects and the interaction techniques used to select those objects can have an impact on task completion time. Besides, testing such interfaces with users in controlled experiments requires considerable time and efforts. To overcome this problem, predictive models, such as the Keystroke-Level Model (KLM), can be used to predict the time required to complete an interactive task at an early stage of the design process. However, so far these models have not been properly extended to address the specific interaction techniques of MR environments. In this paper we propose an extension of the KLM model to interaction performed in MR. First, we propose new operators and experimentally determine the unit times for each of them with a HoloLens v1. Then, we perform experiments based on realistic interaction scenarios to consolidate our model. These experiments confirm the validity of our extension of KLM to predict interaction time in mixed reality environments..","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128756199","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

RNIN-VIO: Robust Neural Inertial Navigation Aided Visual-Inertial Odometry in Challenging Scenes rninvio:鲁棒神经惯性导航辅助视觉惯性里程计在挑战场景

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00043

Danpeng Chen, Nan Wang, Runsen Xu, Weijian Xie, H. Bao, Guofeng Zhang

{"title":"RNIN-VIO: Robust Neural Inertial Navigation Aided Visual-Inertial Odometry in Challenging Scenes","authors":"Danpeng Chen, Nan Wang, Runsen Xu, Weijian Xie, H. Bao, Guofeng Zhang","doi":"10.1109/ismar52148.2021.00043","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00043","url":null,"abstract":"In this work, we propose a tightly-coupled EKF framework for visual-inertial odometry with NIN (Neural Inertial Navigation) aided. Traditional VIO systems are fragile in challenging scenes with weak or confusing visual information, such as weak/repeated texture, dynamic environment, fast camera motion with serious motion blur, etc. It is extremely difficult for a vision-based algorithm to handle these problems. So we firstly design a robust deep learning based inertial network (called RNIN), using only IMU measurements as input. RNIN is significantly more robust in challenging scenes than traditional VIO systems. In order to take full advantage of vision-based algorithms in AR/VR areas, we further develop a multi-sensor fusion system RNIN-VIO, which tightly couples the visual, IMU and NIN measurements. Our system performs robustly in extremely challenging conditions, with high precision both in trajectories and AR effects. The experimental results of evaluation on dataset evaluation and online AR demo demonstrate the superiority of the proposed system in robustness and accuracy.","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"40 7","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114094464","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 17

Keynote Speaker: Wearable Haptics for Virtual and Augmented Reality 主讲人:虚拟和增强现实的可穿戴触觉技术

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00012

引用次数: 0

Evaluating the user Experience of a Photorealistic Social VR Movie 评估一个逼真的社交VR电影的用户体验

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00044

Jie Li, S. Subramanyam, Jack Jansen, Yanni Mei, Ignacio Reimat, K. Lawicka, Pablo César

{"title":"Evaluating the user Experience of a Photorealistic Social VR Movie","authors":"Jie Li, S. Subramanyam, Jack Jansen, Yanni Mei, Ignacio Reimat, K. Lawicka, Pablo César","doi":"10.1109/ismar52148.2021.00044","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00044","url":null,"abstract":"We all enjoy watching movies together. However, this is not always possible if we live apart. While we can remotely share our screens, the experience differs from being together. We present a social Virtual Reality (VR) system that captures, reconstructs, and transmits multiple users’ volumetric representations into a commercially produced 3D virtual movie, so they have the feeling of “being there” together. We conducted a 48-user experiment where we invited users to experience the virtual movie either using a Head Mounted Display (HMD) or using a 2D screen with a game controller. In addition, we invited 14 VR experts to experience both the HMD and the screen version of the movie and discussed their experiences in two focus groups. Our results showed that both end-users and VR experts found that the way they navigated and interacted inside a 3D virtual movie was novel. They also found that the photorealistic volumetric representations enhanced feelings of co-presence. Our study lays the groundwork for future interactive and immersive VR movie co-watching experiences.","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"66 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114650940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

A Reinforcement Learning Approach to Redirected Walking with Passive Haptic Feedback 基于被动触觉反馈的重定向行走的强化学习方法

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00033

Ze-Yin Chen, Yijun Li, Miao Wang, Frank Steinicke, Qinping Zhao

{"title":"A Reinforcement Learning Approach to Redirected Walking with Passive Haptic Feedback","authors":"Ze-Yin Chen, Yijun Li, Miao Wang, Frank Steinicke, Qinping Zhao","doi":"10.1109/ismar52148.2021.00033","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00033","url":null,"abstract":"Various redirected walking (RDW) techniques have been proposed, which unwittingly manipulate the mapping from the user’s physical locomotion to motions of the virtual camera. Thereby, RDW techniques guide users on physical paths with the goal to keep them inside a limited tracking area, whereas users perceive the illusion of being able to walk infinitely in the virtual environment. However, the inconsistency between the user’s virtual and physical location hinders passive haptic feedback when the user interacts with virtual objects, which are represented by physical props in the real environment.In this paper, we present a novel reinforcement learning approach towards RDW with passive haptics. With a novel dense reward function, our method learns to jointly consider physical boundary avoidance and consistency of user-object positioning between virtual and physical spaces. The weights of reward and penalty terms in the reward function are dynamically adjusted to adaptively balance term impacts during the walking process. Experimental results demonstrate the advantages of our technique in comparison to previous approaches. Finally, the code of our technique is provided as an open-source solution.","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128203389","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

Distortion-Aware Room Layout Estimation from A Single Fisheye Image 基于单张鱼眼图像的失真感知房间布局估计

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00061

Ming Meng, Likai Xiao, Yi Zhou, Zhao-Qing Li, Zhong Zhou

{"title":"Distortion-Aware Room Layout Estimation from A Single Fisheye Image","authors":"Ming Meng, Likai Xiao, Yi Zhou, Zhao-Qing Li, Zhong Zhou","doi":"10.1109/ismar52148.2021.00061","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00061","url":null,"abstract":"Omnidirectional images of 180° or 360° field of view provide the entire visual content around the capture cameras, giving rise to more sophisticated scene understanding and reasoning and bringing broad application prospects for VR/AR/MR. As a result, researches on omnidirectional image layout estimation have sprung up in recent years. However, existing layout estimation methods designed for panorama images cannot perform well on fisheye images, mainly due to lack of public fisheye dataset as well as the significantly differences in the positions and degree of distortions caused by different projection models. To fill theses gaps, in this work we first reuse the released large-scale panorama datasets and reproduce them to fisheye images via projection conversion, thereby circumventing the challenge of obtaining high-quality fisheye datasets with ground truth layout annotations. Then, we propose a distortion-aware module according to the distortion of the orthographic projection (i.e., OrthConv) to perform effective features extraction from fisheye images. Additionally, we exploit bidirectional LSTM with two-dimensional step mode for horizontal and vertical prediction to capture the long-range geometric pattern of the object for the global coherent predictions even with occlusion and cluttered scenes. We extensively evaluate our deformable convolution for room layout estimation task. In comparison with state-of-the-art approaches, our approach produces considerable performance gains in real-world dataset as well as in synthetic dataset. This technology provides high-efficiency and low-cost technical implementations for VR house viewing and MR video surveillance. We present an MR-based building video surveillance scene equipped with nine fisheye lens can achieve an immersive hybrid display experience, which can be used for intelligent building management in the future.","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132796040","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

BuildingSketch: Freehand Mid-Air Sketching for Building Modeling BuildingSketch:用于建筑建模的徒手半空素描

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00049

Zhihao Liu, Fanxing Zhang, Zhanglin Cheng

引用次数: 6

Parametric Model Estimation for 3D Clothed Humans from Point Clouds 基于点云的三维穿衣人参数化模型估计

2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Pub Date : 2021-10-01 DOI: 10.1109/ismar52148.2021.00030

Kangkan Wang, Huayu Zheng, Guofeng Zhang, Jian Yang

{"title":"Parametric Model Estimation for 3D Clothed Humans from Point Clouds","authors":"Kangkan Wang, Huayu Zheng, Guofeng Zhang, Jian Yang","doi":"10.1109/ismar52148.2021.00030","DOIUrl":"https://doi.org/10.1109/ismar52148.2021.00030","url":null,"abstract":"This paper presents a novel framework to estimate parametric model- s for 3D clothed humans from partial point clouds. It is a challenging problem due to factors such as arbitrary human shape and pose, large variations in clothing details, and significant missing data. Existing methods mainly focus on estimating the parametric model of undressed bodies or reconstructing the non-parametric 3D shapes from point clouds. In this paper, we propose a hierarchical regression framework to learn the parametric model of detailed human shapes from partial point clouds of a single depth frame. Benefiting from the favorable ability of deep neural networks to model nonlinearity, the proposed framework cascades several successive regression networks to estimate the parameters of detailed 3D human body models in a coarse-to-fine manner. Specifically, the first global regression network extracts global deep features of point clouds to obtain an initial estimation of the undressed human model. Based on the initial estimation, the local regression network then refines the undressed human model by using the local features of neighborhood points of human joints. Finally, the clothing details are inferred as an additive displacement on the refined undressed model using the vertex-level regression network. The experimental results demonstrate that the proposed hierarchical regression approach can accurately predict detailed human shapes from partial point clouds and outperform prior works in the recovery accuracy of 3D human models.","PeriodicalId":395413,"journal":{"name":"2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127440177","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0