Modeling the effect of stimulus perturbations on error correlations between brain and behavior

Over the last decade, machine learning algorithms have proven to be useful tools for exploring neural representations of percepts and concepts in the brain. An important but often neglected next step is it to relate neural representations to human behavior. Here, we introduce a novel approach to definitively linking neural representations to structural properties of stimuli as well as human behavior by analyzing patterns of classification errors using linear mixed-effects (LME) models. An LME model includes a priori predictive models of matching of error patterns between neural decoding and human behavior as fixed effects as well as random effects to account for subject variability. Finally, we demonstrate the viability of this approach using data from a set of fMRI and behavioral experiments testing the influence of visual properties on the neural representation of categories of real-world visual scenes.