Markov Guided Spatio-Temporal Networks for Brain Image Classification*

Abstract

This paper proposes a representation learning model to identify task-state fMRIs for knowledge-concept recognition, which has the potential to model the human cognitive expression system. The traditional CNN-LSTM is usually employed to learn deep features from fMRIs, where CNN aims at extracting the spatial structure and LSTM accounts for the temporal structure. However, the manifold smoothness of the latent features caused by the fMRI sequence is often ignored, leading to unsteady data representation. In this paper, we model latent features as a hidden Markov chain and introduce a Markov-guided Spatio-Temporal Network (MSTNet) for brain image representation. Concretely, MSTNet has three parts: CNN that aims to learn latent features from 3D fMRI frames where a Markov Regularization enforces the neighborhood frames to have similar features, LSTM integrates all frames of an fMRI sequence into a feature vector and fully connected network (FCN) that is to implement the brain image classification. Our model is trained towards minimizing the cross entropy (CE) loss. Our experiment is conducted on the brain fMRI datasets achieved by scanning college students when they were learning five concepts of computer science. The results show that the proposed MSTNet can benefit from the introduced Markov regularization and thus result in improved performance on the brain activity classification. This study not only shows an effective fMRI classification model with Markov regularization but also provides the potential to understand brain intelligence and help patients with language disabilities.