An fMRI-based auditory decoding framework combined with convolutional neural network for predicting the semantics of real-life sounds from brain activity

Abstract

Semantic decoding, understood as predicting the semantic information carried by stimuli presented to subjects based on neural signals, is an active area of research. Previous studies have mainly focused on the visual perception process, with relatively little attention paid to complex auditory decoding. Moreover, simple linear models do not achieve optimal performance for the mapping between brain signals and natural sounds. Therefore, a robust approach that combines a pretrained audio tagging model and a nonlinear multilayer perceptron model was proposed to transfer information from non-invasive measured brain activity to deep learning features, thereby generating sound semantics. The results achieved on previously unseen subjects, training without data from the target subjects, and ultimately predicting natural-sound semantics from the fMRI data of unseen subjects. In the study with 30 subjects, the framework in research achieves 23.21% Top-1 and 51.88% Top-5 accuracy scores, which significantly exceed the baseline scores and the scores of other classical algorithms. The approach advances the decoding of auditory neural excitation with the help of deep neural networks, and the proposed model successfully completes a challenging cross-subject decoding task.