DQN-based Join Order Optimization by Learning Experiences of Running Queries on Spark SQL

Abstract

In a smart grid, various types of queries such as ad-hoc queries and analytic queries are requested for data. There is a limit to query evaluation based on a single node database engines because queries are requested for a large scale of data in the smart grid. In this paper, to improve the performance of retrieving a large scale of data in the smart grid environment, we propose a DQN-based join order optimization model on Spark SQL. The model learns the actual processing time of queries that are evaluated on Spark SQL, not the estimated costs. By learning the optimal join orders from previous experiences, we optimize the join orders with similar performance to Spark SQL without collecting and computing the statistics of an input data set.