Towards making sense of Spark-SQL performance for processing vast distributed RDF datasets

Abstract

Recently, a wide range of Web applications (e.g. DBPedia, Uniprot, and Probase) are built on top of vast RDF knowledge bases and using the SPARQL query language. The continuous growth of these knowledge bases led to the investigation of new paradigms and technologies for storing, accessing, and querying RDF data. In practice, modern big data systems (e.g, Hadoop, Spark) can handle vast relational repositories, however, their application in the Semantic Web context is still limited. One possible reason is that such frameworks rely on distributed systems, which are good for relational data, however, their performance on dealing with graph data models like RDF has not been well-studied yet. In this paper, we present a systematic evaluation of the performance of SparkSQL engine for processing SPARQL queries. We stated it using three relevant RDF relational schemas, and two different storage backends, namely, Hive, and HDFS. In addition, we show the impact of using three different RDF-based partitioning techniques with our relational scenario. Additionally, we discuss the results of our experiments: (i) we present insights about the trade-offs that characterize different experimental configurations, and (ii) we identify the best and the worst ones for the SP2Bench's benchmark scenario.