Bayesian Optimization Based on Pseudo Labels

Abstract

The performance of a machine learning or deep learning algorithm is heavily influenced by its hyperparameters. The selection of the hyperparameters is of great significance. To automatically find a superior-performing set of hyperparameters, Bayesian optimization is a common and effective hyperparameter optimization method. And an early stopping strategy is usually employed in the optimization algorithm to improve efficiency. The early stopped trials cannot run to the end, so their final performance metrics are unavailable. Therefore, the existing Bayesian optimization algorithms fail to use the trials terminated early as samples for modeling. This may result in less information participating in the modeling, which leads to high model uncertainty. In this paper, we propose Bayesian optimization based on pseudo labels (BOPL). We apply the extrapolation of learning curves as the early stopping strategy and the pseudo labels obtainment method. We use the pseudo labels of all trials to model the surrogate model in Bayesian optimization, thereby avoiding the waste of information contained in the early stopped trials. Experiments on the ResNet-18 on the CIFAR-100 dataset show that the proposed BOPL consistently outperforms vanilla Bayesian and Bayesian with early stopping. It proves the effectiveness of the proposed method, which finds better-performing hyperparameters at a faster rate. The proposed method is versatile, conceptually simple, and easy to implement.