A Contextual Semantic Interaction Interface for Virtual Reality Environments

Non-trivial virtual environments often require implementation of complex forms of user interaction, including spatial navigation within the whole environment as well as various forms of interaction with particular components of the environment. Dynamic environments, in which content or context changes in response to user interaction, raise serious problems to both content designers and users. Often in-scene interaction elements are used, but they impede perception of the virtual environment, especially in multi-user scenarios, in which majority of users are passive observers. In this paper, a new approach to navigation and interaction with three-dimensional virtual environments is proposed. This approach uses semantic metadata attached to particular components of the environment to build a dynamic interaction interface on a user's handheld mobile device. The interface is built in a contextual manner, enabling a user to interact with the virtual environment with full awareness of the possible actions. Prototypes of the server and the client modules have been implemented using the Unity engine, and the system has been tested on a large-size Powerwall setup.