Real-time localization and visualization of a sound source for virtual reality applications

In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. This is a part of a larger research effort devoted to studying the effects of synesthesia—the act of experiencing one sense modality as another, e.g., sound as flashes of colors—in virtual reality. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener. We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by the loudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented.