Head-Fingers-Arms: Physically-Coupled and Decoupled Multimodal Interaction Designs in Mobile VR

Abstract

This paper proposes a novel bimanual finger-based gesture called X-Fingers that provides interactive 2D spatial input control using vision-based techniques in mobile VR. This finger-based input modality can be coordinated with the movement of the user's arms or head to provide an additional input modality. The incorporation of the arms or the head provides physically-coupled and physically-decoupled multimodal interactions respectively. Given these two design options, we conducted user studies to understand how the nature of the physical coupling of interactions influences the user's performance and preferences with task consisting of varying degrees of coordination between the modalities. Our results show that physically-decoupled interactions designs are preferred when the degree of coordination is high within the multimodal interaction.