A Bayesian Test for Comparing Classifier Errors

Abstract

Multi-class classification algorithms have become an important tool for the analysis of neuroimaging data. Classification errors contain potentially important information that often goes unreported. It is therefore desirable to quantitatively compare patterns of errors between different experimental conditions. Here we present a Bayesian test that is based on comparing evidence in favor of two competing hypotheses, one stating dependence and one stating independence of two given error patterns. We derive analytical solutions for the likelihoods of both hypotheses. We compare the results from our new test with two other methods of comparing error patterns using data from an fMRI experiment and we substantiate reasons for adopting our proposal and for future work.