Teleport to the Augmented Real-World with Live Interactive Effects (IFX)

Augmented telepresence provides rich communication for people at a distance with interactive blended information between the virtual and real world [Rhee et al. 2017, 2020; Young et al. 2022]. We push the boundaries of augmented telepresence with a novel live media technology, including live capturing, modeling, blending, and interactive effects (IFX) to augment telepresence. Using our technology, people at a distance can connect and communicate with creative storytelling, augmented with novel IFX. We achieve this with the following breakthroughs: 1) digitizing remote spaces and people in real-time, 2) transmitting digitized information across a network, 3) augmenting remote telepresence using real-time visual effects and interactive storytelling with live-blending of 3D virtual assets into the digitized real-world. In this presentation, we will unveil several new technologies and novel IFX that can enrich telepresence, including: • Real-time 360° RGBD video capturing: we will demonstrate capturing 360° RGBD videos using a 360° RGB camera and LiDAR sensor, including synchronization between the RGB and depth streams as well as depth map generation. • IFX with live RGBD videos: we will demonstrate real-time blending of 3D virtual objects into the live 360° RGBD videos, showcasing real-time occlusion and collision handling. • 6-degrees of freedom (DoF) tele-movement: we introduce our recent research [Chen et al. 2022] for volumetric environment capturing and 6-DoF navigation. We will demonstrate real-time navigation (movement and rotation) in captured real surroundings (beyond room scales). We will showcase applications (Figure 1) where we can virtually teleport to and explore within a live stream of the augmented real world and communicate remotely with live IFX.