Relating Deep Neural Network Representations to EEG-fMRI Spatiotemporal Dynamics in a Perceptual Decision-Making Task

Abstract

The hierarchical architecture of deep convolutional neural networks (CNN) resembles the multi-level processing stages of the human visual system during object recognition. Converging evidence suggests that this hierarchical organization is key to the CNN achieving human-level performance in object categorization [22]. In this paper, we leverage the hierarchical organization of the CNN to investigate the spatiotemporal dynamics of rapid visual processing in the human brain. Specifically we focus on perceptual decisions associated with different levels of visual ambiguity. Using simultaneous EEG-fMRI, we demonstrate the temporal and spatial hierarchical correspondences between the multi-stage processing in CNN and the activity observed in the EEG and fMRI. The hierarchical correspondence suggests a processing pathway during rapid visual decision-making that involves the interplay between sensory regions, the default mode network (DMN) and the frontal-parietal control network (FPCN).