Large-Scale Distributed Linkage of Records Containing Spatio-Temporal Information

Abstract

Spatio-temporal information is increasingly made available in modern data sets, together with traditional numerical and categorical attributes. Such information can play a vital role in deciding whether two records, coming from disparate data sources, correspond to the same real-world entity. Linkage of records containing spatio-temporal information requires novel linkage methods and is usually associated with a significant computational overhead. To reduce computational costs, in this paper, we propose the first Spark-based approach for distributed, on-demand, spatio-temporal linkage. Through experimental evaluation, we illustrate that our Spark-based approach achieves (on average) 35% performance improvement compared with the respective Map/Reduce-based implementation.