Sparkle: Deep Learning Driven Autotuning for Taming High-Dimensionality of Spark Deployments

Abstract

The exponential growth of data in the Cloud has highlighted the need for more efficient data processing. In-Memory Computing frameworks (e.g., Spark) offer improved efficiency for large-scale data analytics, however, they also provide a plethora of configuration parameters that affect the resource consumption and performance of applications. Manually optimizing these parameters is a time-consuming process, due to i) the high-dimensional configuration space, ii) the complex inter-relationship between different parameters, iii) the diverse nature of workloads and iv) the inherent data heterogeneity. We introduce Sparkle , an end-to-end deep learning-based framework for automating the performance modeling and tuning of Spark applications. We introduce a modular DNN architecture that expands to the entire Spark parameter configuration space and provides a universal performance modeling approach, completely eliminating the need for human or statistical reasoning. By employing a genetic optimization process, Sparkle quickly traverses the design space and identifies highly optimized Spark configurations. Our experiments on the HiBench benchmark suite show that Sparkle delivers an average prediction accuracy of 93%, with high generalization capabilities, i.e., $\approx 80\%$ accuracy for unseen workloads, dataset sizes and configurations, outperforming state-of-art. Regarding end-to-end optimization, Sparkle efficiently explores Spark's high-dimensional parameter space, delivering new dominant Spark configurations, which correspond to 65% Pareto coverage w.r.t its Spark native optimization counterpart.