A Comparative Analysis of Hyperopt as Against Other Approaches for Hyper-Parameter Optimization of XGBoost

Abstract

The impact of Hyper-Parameter optimization on the performance of a machine learning algorithm has been proved both theoretically and empirically by many studies reported in the literature. It is a tedious and a time-consuming task if one goes for Manual Search. Some of the common approaches to address this include Grid search and Random search. Another alternative is performing the Bayesian optimization using the Hyperopt library in Python. In this paper, we tune the hyperparameters of XGBoost algorithm on six real world datasets using Hyperopt, Random search and Grid Search. We then compare the performances of each of these three techniques for hyperparameter optimization using both accuracy and time taken. We find that the Hyperopt performs better than the Grid search and Random search approaches taking into account both accuracy and time. We conclude that Bayesian optimization using Hyperopt is the most efficient technique for hyperparameter optimization.