Evaluation of Multiple Apache Spark Applications using Kubernetes as a Cluster manager on Google Cloud

Abstract

Big data processing frameworks demands for scalable and efficient cluster management. Apache Spark has emerged as prominent big data processing framework providing high-speed data processing and analytics capabilities for multiple applications. This paper explores the integration of Kubernetes as a cluster manager for Apache Spark applications leveraging its containerization capabilities to improve resource utilization and simplify deployment. In this paper the challenges of deploying spark applications on traditional cluster managers and showcase the advantages of adopting Kubernetes are analysed. The experimental evaluation demonstrates the benefits of Kubernetes as a cluster manager for Apache Spark framework. To execute the multiple Apache Spark applications on Kubernetes a homogenous cluster on Google Cloud is created by History bucket and service account. Finally multiple applications are executed on Google Kubernetes Engine. Output can be shown as the number of executor pods created with the performance metrics can be viewed. In conclusion, this paper compares the performance metrics such as job execution time and resource utilization with the different cluster.