Multi-finger AR typing interface for mobile devices

Abstract

In this paper, we propose a user interface that enables multi-finger typing in the space behind a mobile device. By using the augmented reality (AR) technology, a virtual keyboard is superimposed on the rear camera image, and a hand region of the camera image is again superimposed on that image, which makes it possible to perform input operation as if there were a real keyboard. The system recognizes only key pressing actions and does not recognize a hand or fingers, which enables stable recognition and multi-finger input. Further, key typing at any place on a plane and in the air is possible. Demonstration using an experimental device showed that multi-finger input using a virtual keyboard displayed on the screen was realized.