Understanding the Behavior of Spark Workloads from Linux Kernel Parameters Perspective

Despite a number of innovative computer systems with high capacity memory have been built, the design principles behind an operating system kernel have remained unchanged for decades. We argue that kernel parameters is a kind of special interface of operating system and must be factored into the operation and maintenance of datacenters. To shed some light on the effectiveness of tuning Linux parameters of virtual memory subsystem when running Spark workloads, we evaluate the benchmarks in a simple standalone deploy mode. Our performance results reveal that some of the Linux memory parameters must be carefully set to efficiently support these processing workloads. We hope this work yields insights for datacenter system operators.