How different kinds of sound in videos can influence gaze

Abstract

This paper presents an analysis of the effect of thirteen different kinds of sound on visual gaze when looking freely at videos to help to predict eye positions. First, an audio-visual experiment was designed with two groups of participants, with audio-visual (AV) and visual (V) conditions, to test the sound effect. Then, an audio experiment was designed to validate the classification of sound we proposed. We observed that the sound effect is different depending on the kind of sound, and that the classes with human voice (speech, singer, human noise and singers) have the greatest effect. Finally, a comparison of eye positions with a visual saliency model was carried out, which proves that adding sound to video decreases the accuracy of prediction of the visual saliency model.