Behind-the-mask: a face-through head-mounted display

Abstract

A head-mounted display (HMD), which is common in virtual reality (VR) systems, normally hides the user's face. This feature prohibits to realize a face-to-face communication, in which two or more users share the same virtual space, or show a participant's face on a surrogate-robot's face when the user remotely connects to the robot through an HMD for tele-immersion. Considering that face-to-face communication is one of the fundamental requirements of real-time communications, and is widely realized and used by many nonVR telecommunication systems, an HMD's face hiding feature is considered to be a serious problem and limits the possibility of VR. To address this issue, we propose the notion of "Face-through HMD" and present a face-capturing HMD configuration called "Behind-the-Mask" with infrared (IR) cut filters and side cameras that can be attached to existing HMDs. As an IR cut filter only reflects infrared light and transmits visible light, it is transparent to the user's eye but reflects the user's face with infrared lights. By merging a prescanned 3D face model of the user with the face image obtained from our HMD, the 3D face model of the user with eyes and mouth movement can be reconstructed. We consider that our proposed HMD can be used in many VR applications.