Sparse Bayesian learning and the relevance multi-layer perceptron network

Abstract

We introduce a simple framework for sparse Bayesian learning with multi-layer perceptron (IMLP) networks, inspired by Tipping's relevance vector machine (RVM). Like the RVM, a Bayesian prior is adopted that includes separate hyperparameters for each weight, allowing redundant weights and hidden layer units to be identified and subsequently pruned from the network, whilst also providing a means to avoid over-fitting the training data. This approach is also more easily implemented, as only the diagonal elements of the Hessian matrix are used in the update formula for the regularisation parameters, rather than the traces of square sub-matrices of the Hessian corresponding to the weights associated with each regularisation parameter. The proposed relevance multi-layer perceptron (RMLP) is evaluated over several publicly available benchmark datasets, demonstrating the viability of the approach, giving rise to similar generalisation performance, but with far fewer weights.