CoBABO: A Hyperparameter Search Method with Cost Budget Awareness

Abstract

In AutoML, Bayesian optimization (BO) is commonly used to automatically search for the hyperparameters that yield optimal model performance. Since an essential step in BO, namely model evaluation, is usually very costly in terms of computation time, some cost-aware BO methods appeared in the literature. The basic idea of these cost-aware methods is to maximize the expected improvement (EI) of model performance per unit of cost at each step. However, these works either do not consider the cost budget or still give more opportunities to low-cost hyperparameters even when the remaining budget runs low. This paper introduces a cost budget aware BO (CoBABO), which goes more aggressively after the hyperparameters that yield higher EI when the remaining cost budget becomes smaller. Experimental results on different machine learning models show that CoBABO often finds significantly better performing models within budget than the aforementioned cost-aware methods do.