Audiovisual stimulus-driven contributions to spatial orienting in ecologically valid situations: An fMRI study

Abstract

Mechanisms of audiovisual attention have been extensively investigated, yet little is known about their functioning in ecologically-valid situations. Here, we investigated brain activity associated with audiovisual stimulus-driven attention using naturalistic stimuli. We created 120 short videos (2.5 s) showing scenes of everyday life. Each video included a visual event comprising a lateralized (left/right) increase in visual saliency (e.g., an actor moving an object), plus a co-occurring sound either on the same or the opposite side of space. Subjects viewed the videos with/without the associated sounds, and either in covert (central fixation) or overt (eye-movements allowed) viewing conditions. For each stimulus, we used computational models (‘saliency maps’) to characterize auditory and visual stimulus-driven signals, and eye-movements (recorded in free viewing) as a measure of the efficacy of these signals for spatial orienting. Results showed that visual saliency modulated activity in the occipital cortex contralateral to the visual event; while auditory saliency modulated activity in the superior temporal gyrus bilaterally. In the posterior parietal cortex activity increased with increasing auditory saliency, but only when the auditory stimulus was on the same side as the visual event. The efficacy of the stimulus-driven signals modulated activity in the visual cortex. We conclude that: (1) audiovisual attention can be successfully investigated in real-like situations; (2) activity in sensory areas reflects a combination of stimulus-driven signals (saliency) and their efficacy for spatial orienting; (3) posterior parietal cortex processes auditory input as a function of its spatial relationship with the visual input.