Information metrics for model selection in function estimation

Abstract

A model selection framework is presented for function estimation under limited information, where only a small set of (noisy) data points are available for inferring the nonconvex unknown function of interest. The framework introduces information-theoretic metrics which quantify model complexity and are used in a multi-objective formulation of the function estimation problem. The intricate relationship between information obtained through observations and model complexity is investigated. The framework is applied to the hyperparameter selection problem in Gaussian Process Regression. As a result of its generality, the framework introduced is applicable to a variety of settings and practical problems with information limitations such as channel estimation, black-box optimisation, and dual control.