Decoding Semantics Categorization during Natural Viewing of Video Streams

Abstract

Exploring the functional mechanism of the human brain during semantics categorization and subsequently leverage current semantics-oriented multimedia analysis by functional brain imaging have been receiving great attention in recent years. In the field, most of existing studies utilized strictly controlled laboratory paradigms as experimental settings in brain imaging data acquisition. They also face the critical problem of modeling functional brain response from acquired brain imaging data. In this paper, we present a brain decoding study based on sparse multinomial logistic regression (SMLR) algorithm to explore the brain regions and functional interactions during semantics categorization. The setups of our study are two folds. First, we use naturalistic video streams as stimuli in functional magnetic resonance imaging (fMRI) to simulate the complex environment for semantics perception that the human brain has to process in real life. Second, we model brain responses to semantics categorization as functional interactions among large-scale brain networks. Our experimental results show that semantics categorization can be accurately predicted by both intrasubject and intersubject brain decoding models. The brain responses identified by the decoding model reveal that a wide range of brain regions and functional interactions are recruited during semantics categorization. Especially, the working memory system exhibits significant contributions. Other substantially involved brain systems include emotion, attention, vision and language systems.