Optimal Bayesian feature selection

Abstract

Biomarker discovery and classification in medical applications both typically involve feature selection applied to a small-sample high-dimensional dataset. Recent work has proposed a framework to integrate a prior over an uncertainty class of parameterized feature-label distributions with training data to obtain optimal classifiers, MMSE classifier error estimates, and evaluate the MSE of error estimates. However, feature selection has not been investigated rigorously in this paradigm. In the present work, we begin to address optimal feature selection in a Bayesian framework via a sparsity inducing prior that assumes the number of “good” features is small. From modeling assumptions and this prior we derive expressions for the sample-conditioned probability mass over good feature sets. It thus becomes possible to find feature sets that are optimal relative to maximal posterior probability. Furthermore, one may provide this probability along with a given feature set, and thereby evaluate the validity and reliability of the results.