Decoding Categories from Human Brain Activity in the Human Visual Cortex Using the Triplet Network

Abstract

Decoding visual stimuli from functional magnetic resonance imaging (fMRI) is of great significance for understanding the neural mechanism of the visual information processing in the human brain. How to extract effective information from massive voxel data in the brain to predict the brain state is a problem worth discussing in fMRI. However, the inherent characteristics of small quantity and high dimensionality in fMRI data limited the performance of brain decoding. As an effective way to acquire visual information, people usually compare with the prior knowledge learned when recognizing objects, and does not need to have a complete understanding of visual information. In this paper, we proposed a new visual classification model to decode the stimulus categories from the visual information of the brain based on the triplet network. The triplet network is a model framework with a comparison mechanism similar to that of human visual recognition objects, contains three-branches weight sharing subnetworks, which are composed of fully connected networks in our model. Our results showed that the decoding accuracy is 57.5±1.86% and 44.17±1.31% for S1 and S2, respectively. S1 was about 6% higher than the best traditional machine learning classifier SVM, while S2 was nearly 3.5% higher than SVM. Our results fully confirmed the validity of comparing the differences between samples in fMRI data with small quantity.