MuSS-bits: sensor-display blocks for deaf people to explore musical sounds

Abstract

Hearing loss makes learning a musical instrument a challenging task. Prior work suggests that a universal sensory substitution system that works uniformly across all deaf users may not exist given the diversity within the deaf community. In this paper, we present Music Sensory Substitution (MuSS) Bits, wireless sensor-display pairs that enable exploration of musical sound as well as customization of visual and vibrotactile feedback to cater to individual requirements and preferences. MuSS-Bits are portable, easy to deploy on the user's body, on an instrument, or in the environment, and provide real-time feedback. We review existing music sensory substitution systems, discuss the design space for MuSS-Bits, present details of a prototypical implementation and illustrate interaction possibilities including initial user reactions.