Informing the Use of Hyperparameter Optimization Through Metalearning

One of the challenges of data mining is finding hyperparameters for a learning algorithm that will produce the best model for a given dataset. Hyperparameter optimization automates this process, but it can still take significant time. It has been found that hyperparameter optimization does not always result in induced models with significant improvement over default values, yet no systematic analysis of the role of hyperparameter optimization in machine learning has been conducted. We use metalearning to inform the decision of whether to optimize hyperparameters based on expected performance improvement and computational cost.