Dynamic HDR environment capture for mixed reality

Abstract

Rendering accurate and convincing virtual content into mixed reality (MR) scenes requires detailed illumination information about the real environment. In existing MR systems, this information is often captured using light probes [1, 8, 9, 17, 19--21], or by reconstructing the real environment as a preprocess [31, 38, 54]. We present a method for capturing and updating a HDR radiance map of the real environment and tracking camera motion in real time using a self-contained camera system, without prior knowledge about the real scene. The method is capable of producing plausible results immediately and improving in quality as more of the scene is reconstructed. We demonstrate how this can be used to render convincing virtual objects whose illumination changes dynamically to reflect the changing real environment around them.