Where do I go? Decoding temporal neural dynamics of scene processing and visuospatial memory interactions using convolutional neural networks.

Abstract

Visual scene perception enables rapid interpretation of the surrounding environment by integrating multiple visual features related to task demands and context, which is essential for goal-directed behavior. In the present work, we investigated the temporal neural dynamics underlying the interaction between the processing of bottom-up visual features and top-down contextual knowledge during scene perception. We asked whether newly acquired spatial knowledge would immediately modulate the early neural responses involved in the extraction of navigational affordances available (i.e., the number of open doors). For this purpose, we analyzed electroencephalographic data from 30 participants performing interleaved blocks of a scene memory task and a visuospatial memory task in which we manipulated the number of navigational affordances available. We used convolutional neural networks coupled with gradient-weighted class activation mapping to assess the main electroencephalographic channels and time points contributing to the classification performances. The results indicated an early temporal window of integration in occipitoparietal activity (50-250 ms post stimulus) for several aspects of visual perception, including scene color and number of affordances, as well as for spatial memory content. Moreover, a convolutional neural network trained to detect affordances in the scene memory task failed to generalize to detect the same affordances after participants learned spatial information about goal position within the scene. Taken together, these results reveal an early common window of integration for scene and visuospatial memory information, with a specific and immediate top-down influence of newly acquired spatial knowledge on early neural correlates of scene perception.