A Decision-Theoretic Approach to Model Choice

Abstract

Model choice algorithms are usually compared based on their accuracy, i.e. ability to find true models. However, conservative algorithms (such as BIC minimisation) are accurate when no true effects exist, while more liberal algorithms (such as Lasso) are accurate when there are plenty of true effects. There is ambiguity, then, regarding the correct algorithm. The purpose of this paper is to show how expected utility maximisation and Monte Carlo simulations can be used to compare model choice algorithms. Two loss functions are derived from the expected utility function of the researcher. Both loss functions turn out to be linear combinations of specificity and one or two kinds of sensitivity which are discussed in this paper. Subsequently, this paper experiments with four parametrisations of these loss functions, and then uses these parametrised versions to compare nine algorithms within the contexts of both logistic and Gaussian regression. The results demonstrate that researchers who avoid false positives should either use BIC or BICc for model choice or report nothing at all. AIC does not seem to be the optimal method for the range of parameters covered in this study.