See-through mobile AR system for natural 3D interaction

In this paper, we propose an interaction system which displays see-through images on the mobile display and that allows a user to interact with virtual objects overlaid on the see-through image using the user's hand. In this system, the camera which tracks the user's viewpoint is attached to the front of the mobile display and the depth camera which captures color and depth images of the user's hand and the background scene is attached to the back of the mobile display. Natural interaction with virtual objects using the user's hand is realized by displaying images so that the appearance of a space through the mobile display is consistent with that of the real space from the user's viewpoint. We implemented two applications to the system and showed the usefulness of this system in various AR applications.