Prototyping natural interactions in virtual studio environments by demonstration: combining spatial mapping with gesture following

A virtual studio enables the real-time combination of people or other real objects with computer generated environments. In this paper, a rapid prototyping strategy for natural real-time interactions in such virtual environments is introduced. Its name is ANID, for Authoring Natural Interactions by Demonstration, and focuses on tracking an actor and manipulating control parameters as a way of steering interactions. ANID has two main aspects. The first one is the authoring of spatial relationships between the actor and the virtual environment. The second aspect is the use of gesture following to synchronize various animations or event sequences inside the virtual environment to the corresponding actor's movements. Both aspects allow for a programming-by-demonstration approach, enabling developers to rapidly create the desired interactions by providing examples directly inside the blue or green box of the virtual studio. The methods and tools supporting the strategy are presented. A test case has been developed to demonstrate the applicability of the strategy, in which an interactive virtual robotic arm was created, that could be controlled by hand movements in a natural manner. The advantages and shortcomings of ANID are discussed based on this test case.