Dynamic adaptive mesh streaming for real-time 3D teleimmersion

Abstract

Recent advances in full body 3D reconstruction methods have lead to the realisation of high quality, real-time, photo realistic capture of users in a range of tele-immersion (TI) contexts including gaming and mixed reality environments. The full body reconstruction (FBR) process is computationally expensive requiring comparatively high CPU, GPU and network resources in order to maintain a shared, virtual reality in which high quality 3D reproductions of users can be rendered in real-time. A significant optimisation of the delivery of FBR content has been achieved through the real-time compression and de-compression of 3D geometry and textures. Here we present a new, adaptive compression methodology that allows a TI system called 3D-LIVE to modify the quality and speed of a FBR TI pipeline based on the data carrying capability of the network. Our rule-based adaptation strategy uses network performance sampling processes and a configurable rule engine to dynamically alter the compression of FBR reconstruction on-the-fly. We demonstrate the efficacy of the approach with an experimental evaluation of system and conclude with a discussion of future directions for adaptive FBR compression.