Mobile Cross Reality (XR) space for remote collaboration

In this study, we propose an XR (cross reality) dialogue system that transmits an omnidirectional stereoscopic moving viewpoint image of a remote real space and presents it to a local worker with an HMD (Head Mounted Display), and the worker faces the stereoscopic avatar (face part) of the local worker presented with MR glasses. The system is asymmetrical.This system is asymmetric. The position and orientation of the head and eyes measured by the HMD (Vive Pro Eye, HTC) at the local site are transmitted to the remote space, and the avatar of the local worker is shown by the MR glasses (Magic Leap, HoloLens2). The CG object and avatar (the face of the local worker) are shared with the remote 3D real space. This enables the remote worker to see the face of the local worker who has the viewpoint position on the TwinCam Go mobile stereoscopic camera.We conducted an experiment to evaluate the reality of the avatars presented in the MR glasses of the field workers as interactors and the clarity of the instructions for the spatial objects to be discussed. The participants in the experiment were seven university (graduate) students (aged 21-24). The communication time of this system was about 70 ms one way. The G.1010 of ITU-T recommends that the delay between terminals for real-time video communication should be less than 150 ms one way.The experiment participants stood in front of the stereoscopic camera (TwinCam) as remote experiencers, and wore MR glasses to see the avatars of the local experiencers (wearing HMDs). The local operator describes three spatial objects (a cube, a sphere, and a cone) that are displayed 1.7 m closer to the experimental participant from the remote camera.Analysis of variance showed that the reality and clarity of the dialogue increased at the 1% significance level when the avatar's head rotation and eye movements were present. The 3D projection display was higher than 2D at the 5% level of significance. The clarity of the subject improved at the 5% level of significance when there was head rotation and eye movement, but the 3D projection was not significantly different from the 2D.This indicates that the 2D monitor without binocular disparity has a large reading error in the depth direction. Another clue is that the HMD and MR glasses can utilize motion parallax by moving the head even slightly. This makes it possible to reduce the error to less than about half that of a 2D monitor.In this study, we developed a cross reality (XR) dialogue system for mobile remote collaboration. As a result of three kinds of evaluation experiments, it was shown that the clarity of dialogue and the accuracy of depth indication were improved compared with the condition of the conventional 2D video teleconference. This system can also be integrated with the metaverse, and can provide a variety of remote experiences of the world even when mobility is restricted due to current contagious disease. In the future, we will continue to verify and demonstrate the technology necessary for smooth telecommunication.