Performance Study of Spindle, A Web Analytics Query Engine Implemented in Spark

Abstract

This paper shares our experiences building and benchmarking Spindle as an open source Spark-based web analytics platform. Spindle's design has been motivated by real-world queries and data requiring concurrent, low latency query execution. We identify a search space of Spark tuning options and study their impact on Spark's performance. Results from a self-hosted six node cluster with one week of analytics data (13.1GB) indicate tuning options such as proper partitioning can cause a 5x performance improvement.