Structured interaction in the SoundThimble real-time gesture sonification framework

Abstract

We introduce SoundThimble, a design platform for layered sonic interaction based on the relationship between human motion and virtual objects in 3D space. A Vicon motion capture system and custom software are used to track, interpret and sonify the movement and gestures of a performer relative to a virtual object. We define three possible interaction dynamics, centred around object search, manipulation and arrangement. We explore the resulting possibilities for layered structures and extended perception and expression. The software developed is open source and portable to similar hardware systems, leaving room for further extension of the interaction mechanics.