Confluence: Adaptive Spatiotemporal Data Integration Using Distributed Query Relaxation over Heterogeneous Observational Datasets

Abstract

Combining data from disparate sources enhances the opportunity to explore different aspects of the phenomena under consideration. However, there are several challenges in doing so effectively that include, inter alia, the heterogeneity in data representation and format, collection patterns, and integration of foreign data attributes in a ready-to-use condition. In this study, we have designed a scalable query-oriented distributed data integration framework, Confluence, that also dynamically generates accurate interpolations for the targeted spatiotemporal scopes along with an estimate of the uncertainty involved with such estimation in case of spatiotemporal misalignment of datapoints. Confluence efficiently orchestrates computations to evaluate spatiotemporal query joins and facilitates distributed query evaluations with a dynamic relaxation of query constraints. Query evaluations are locality-aware and we leverage model-based dynamic parameter selection to provide accurate estimation for data points. We have included empirical benchmarks that profile our system in terms of accuracy, latency, and throughput at scale and also demonstrate its improvement in performance in a distributed cloud computing environment over GeoMesa, a Spark-based geospatial analytics framework.