Scalable Analysis of Open Data Graphs

Abstract

We have studied Open Data as a connected graph. Each data package is considered a vertex, and we studied the similarity graph induced by several different similarity measures. We analyzed the resulting similarity graph using different metrics to estimate its quality and informativeness. In order to cope with the size of the open data graph (over 6 billion edges), the graph constructions and analysis are done using a distributed computation framework, Apache Spark. The algorithms were implemented using the Spark resilient distributed data algebra, and executed on the Google Cloud Platform (GCP).