Spark and Scala (keynote)

Abstract

Written mostly in Scala and with over 1000 contributors, Apache Spark has become the de facto standard for big data processing. In this talk, I will review the evolution of Spark for the last seven years and our experience using Scala as the main programming language in a high profile open source project with a distributed team. I will outline language features that we can't live without, and features we wish were designed differently. Last but not least, I will discuss how we at Databricks are leveraging native code to further improve performance.