Unraveling Audiovisual Perception Across Space and Time: A Neuroinspired Computational Architecture

Abstract

Accurate perception of audiovisual stimuli depends crucially on the spatial and temporal properties of each sensory component, with multisensory enhancement only occurring if those components are presented in spatiotemporal congruency. Although spatial localization and temporal detection of audiovisual signals have each been extensively studied, the neural mechanisms underlying their joint influence, particularly in spatiotemporally misaligned contexts, remain poorly understood. Moreover, empirical dissection of their respective contributions to behavioral outcomes proves challenging when spatial and temporal disparities are introduced concurrently. Here, we sought to elucidate the mutual interaction of temporal and spatial offsets on the neural encoding of audiovisual stimuli. To this end, we developed a biologically inspired neurocomputational model that reproduces behavioral evidence of perceptual phenomena observed in audiovisual tasks, i.e., the modality switch effect (temporal realm) and the ventriloquist effect (spatial realm). Tested against the race model, our network successfully simulates multisensory enhancement in reaction times due to the concurrent presentation of cross-modal stimuli. Further investigation on the mechanisms implemented in the network upheld the centrality of cross-sensory inhibition in explaining modality switch effects and of cross-modal and lateral intra-area connections in regulating the evolution of these effects in space. Finally, the model predicts an amelioration in temporal detection of different modality stimuli with increasing between-stimuli eccentricity and indicates a plausible reduction in auditory localization bias for increasing interstimulus interval between spatially disparate cues. Our findings provide novel insights into the neural computations underlying audiovisual perception and offer a comprehensive predictive framework to guide future experimental investigations of multisensory integration.