Dynamic Fidelity Selection for Hyperparameter Optimization

The dramatic growth of deep learning over the past decade has increased the demand for effective hyperparameter optimization (HPO). At the moment, evolutionary algorithms such as the covariance matrix adaptation evolution strategy (CMA-ES) are recognized as one of the most promising approaches for HPO. However, it is often problematic for practitioners that HPO is a time-consuming task because of its computationally expensive objective even if evaluations were parallelized in each generation of an evolutionary algorithm. To address the problem, multi-fidelity optimization that exploits cheap-to-evaluate lower-fidelity alternatives instead of the true maximum-fidelity objective can be utilized for faster optimization. In this paper, we introduce a new fidelity-selecting strategy designed to solve HPO problems with an evolutionary algorithm. Then, we demonstrate that the CMA-ES with the proposed strategy accelerates the search by about 8.5%--15% compared with the vanilla CMA-ES while keeping the quality of the solutions obtained.