Improving Spark performance with zero-copy buffer management and RDMA

Abstract

With the ever increasing demand on interactive data analytics, latency for big data frameworks becomes more important. We present our preliminary experience designing and implementing NetSpark, an improved Spark [1] framework that is highly optimized for network latency. Combining optimizations on data serialization, network buffer management with hardware-supported Remote Direct Memory Access (RDMA) technology, we show that we can eliminate most of the data copies from end to end, significantly reducing the Spark task running time. Our preliminary experiments show that NetSpark improves GroupBy operation in Spark by about 40% and the PageRank algorithm in GraphX by about 20% on a 10Gbps data center network over the legacy network stack.