Blending Shadows: Casting Shadows in Virtual and Real using Occlusion-Capable Augmented Reality Near-Eye Displays

The fundamental goal of augmented reality (AR) is to integrate virtual objects into the user’s perceived reality seamlessly. However, various issues hinder this integration. In particular, Optical See Through (OST) AR is hampered by the need for light subtraction due to its see-through nature, making some basic rendering harder to realize. In this paper, we realize mutual shadows between real and virtual objects in OST AR to improve this virtual-real integration. Shadows are a classic problem in computer graphics, virtual reality, and video see-through AR, yet they have not been fully explored in OST AR due to the light subtraction requirement. We build a proof-of-concept system that combines a custom occlusion-capable OST display, global light source estimation, 3D registration, and ray-tracing-based rendering. We will demonstrate mutual shadows using a prototype and demonstrate its effectiveness by quantitatively evaluating shadows with the real environment using a perceptual visual metric.