On the Auto-Tuning of Elastic-search based on Machine Learning

Abstract

Elastic-search is a distributed search engine which is used to process large amount of data widely. It has a vast number of configuration parameters which are extremely difficult to manually tune to achieve optimal throughput and latency. This paper presents an auto-tuning method to improve the performance of Elastic-search based on random forest and gradient boosting regression trees. By analyzing the working process of Elastic-search, performance-sensitive configuration parameters are selected to establish a machine learning model with high accuracy, so as to accurately predict the performance of Elastic-search with different configurations. With the help of performance prediction, the genetic algorithm finds the optimal configuration of Elastic-search under given system conditions. Three data sets with different sizes and structures are selected for evaluation and the benchmarking tool EsRally tests the performance of index and query operation. Experimental results show that our proposed method can improve the performance by 2.73 times on average and up to 7.02 times compared to the default configuration of Elastic-search.