Effect of Visibility of Auditory Stimulus Location on Ventriloquism Effect using AR-Head-Mounted Display

Abstract

Virtual reality (VR) or augmented reality (AR) games using head-mounted displays (HMDs) are becoming increasingly popular in recent games. These games can present wider visual stimuli than TVs or handheld games. Moreover, the location of auditory stimuli is presented to the same location as visual stimuli. Therefore, we propose presenting visual stimuli to auditory stimuli rather than presenting auditory stimuli to visual stimuli. When we present visual stimuli to auditory stimuli, it is necessary to clarify how far the locations of the sound source and visual stimuli can be shifted. Thus, we examined varying degrees of spatial disparity between auditory and visual stimuli to determine whether they are still perceived as originating from the same location. The ventriloquism effect is known as a cross-modality between the locations of auditory and visual stimuli. Many researchers investigated the ventriloquism effect; however, there is no research on the effect of visibility of a loudspeaker playing a sound on the ventriloquism effect. In this study, we aim to clarify the effect of visibility of a loudspeaker playing a sound on the ventriloquism effect. For this purpose, we conducted two experiments to determine whether auditory and visual stimuli are originating from the same location when there are varying degrees of spatial disparity between them and measure their angle of origin. One was an AR condition experiment in which measurements were made with the loudspeaker visible, whereas the other was a VR condition experiment in which the loudspeaker was not visible. From the experimental results, the discrimination threshold of angle was more significant under the condition in which the loudspeaker was visible (AR condition experiment) than under the condition in which the loudspeaker was not visible (VR condition experiment). The results show that the ventriloquism effect is more substantial when the loudspeaker is visible.