MiLa: An Audiovisual Instrument for Learning the Curwen Hand Signs

Abstract

We present a novel musical system for learning the Curwen-Kodaly hand signs, which stand for solfege syllables and pitches (do, re, mi, etc.). These hand postures comprise a system of musical notation typically used in sight-singing training to familiarize with patterns of musical tones. Therefore, what concerns us is music literacy as opposed to practice with any particular instrument. Our system uses a Leap motion sensor to recognize the hand signs and produce corresponding user-recorded sounds. For each hand sign, the system also displays a natural 3D scenery, which is intended to match the metaphorical nature of the tone and has a resemblance to the hand posture (e.g. mi is a steady or calm tone, so the hand sign is an open hand with palm downwards and the scenery is a calm but moving ocean).