KORDI: A Framework for Real-Time Performance and Cost Optimization of Apache Spark

Apache Spark is one of the most commonly used frameworks for Big Data processing. Research on the provided streaming dynamic resource allocation feature has shown that large data load fluctuations, such as those seen in website traffic, negatively impact automatic scaling. Further analysis indicates that the lack of data load prediction, particularly during peak hours or days, is the root cause of this issue. In this paper, we propose an enhanced solution, KORDI (Knowledge-based Orchestrated Resource Distribution), which enables the online modeling of data size for real-time forecasting and control of future Spark resource requirements. This solution optimizes the allocation of Spark resources for streaming applications by employing a well-established machine learning technique, namely SARIMAX, to forecast anticipated data load before passing it to the Load Balancing Engine. The engine then appropriately adjusts the Spark cluster size. Experimental results demonstrate that the proposed solution reduces costs by 38% compared to static resource provisioning without compromising stability.