Reproduction of Environment Reflection using Extrapolation of Front Camera Images in Mobile AR

In this paper, we propose a method to reproduce the reflection of a real scene on a virtual object using only the images captured by a camera attached to the front of a mobile device. Since it is not possible to acquire the entire scene using only the front camera, the area surrounding the front camera image is extrapolated to obtain a sufficient scene for reflection. Image transformation using deep neural networks is used for extrapolation, and high-quality and stable extrapolation is realized by using the extrapolation results in the previous frame. As a result of experiment that evaluated the quality of extrapolation and reproduction of reflection, we confirmed that both extrapolated images and reproduced reflection looked natural. We also conducted an experiment to evaluate users' impression of the reflection. The results showed that the proposed method was effective for participants who knew about AR in naturalness of reflection and material perception.