Towards a linguistically motivated model for selection in virtual reality

Swiftness and robustness of natural communication is tied to the redundancy and complementarity found in our multimodal communication. Swiftness and robustness of human-computer interaction (HCI) is also a key to the success of a virtual reality (VR) environment. The interpretation of multimodal interaction signals has therefore been considered a high goal in VR research, e.g. following the visions of Bolt's put-that-there in 1980 [1]. It is our impression that research on user interfaces for VR systems has been focused primarily on finding and evaluating technical solutions and thus followed a technology-oriented approach to HCI. In this article, we argue to complement this by a human-oriented approach based on the observation of human-human interaction. The aim is to find models of human-human interaction that can be used to create user interfaces that feel natural. As the field of Linguistics is dedicated to the observation and modeling of human-human communication, it could be worthwhile to approach natural user interfaces from a linguistic perspective. We expect at least two benefits from following this approach. First, the human-oriented approach substantiates our understanding of natural human interactions. Second, it brings about a new perspective by taking the interaction capabilities of a human addressee into account, which are not often explicitly considered or compared with that of the system. As a consequence of following both approaches to create user interfaces, we expect more general models of human interaction to emerge.