Voice and movement as predictors of gesture types and physical effort in virtual object interactions of classical Indian singing

This paper reports on the exploration of the relationships between gesture and sound in the context of practicing 'sound sculpting' [4] in Hindustani (Dhrupad) vocal improvisation. In this practice, singers of the classical Indian music tradition often engage with melodic ideas by manipulating intangible, imaginary objects and materials with their hands while singing. Here we explore the interaction possibilities that both malleable (through elasticity) as well as rigid (through weight/friction) objects can afford by accounting for the physical effort that these require. Specifically, we focus on using movement and audio features for (a) predicting bodily effort levels through linear regression and (b) classifying gestures as either elastic or rigid interactions through logistic regression. The results suggest that a good part of the variance in both physical effort and gesture type can be explained through a small set of audio and motion features.