Wearable sensors for real-time musical signal processing

Abstract

This paper describes the use of wearable sensor technology to control parameters of audio effects for real-time musical signal processing. Traditional instrument performance techniques are preserved while the system modifies the resulting sound based upon the movements of the performer. Gesture data from a performing artist is captured using three-axis accelerometer packages that is converted to MIDI (musical instrument digital interface) messages using microcontroller technology. ChucK, a new programming language for on-the-fly audio signal processing and sound synthesis, is used to collect and process synchronized gesture data and audio signals from the traditional instrument being performed. Case studies using the wearable sensors in a variety of locations on the body (head, hands, feet, etc.) with a number of different traditional instruments (tabla, sitar, drumset, turntables, etc.) are presented.