Scalable Spatial Analytics and In Situ Query Processing in DaskDB

Abstract

Vast amounts of data are stored in raw data files. Data scientists and practitioners typically use data science frameworks for data analysis on raw data. Among them, Python Pandas library is one of the most popular language-based frameworks. On the other hand, relational databases (RDBMSs) are still widely used for SQL query execution. Before querying, raw data must be loaded into RDBMSs through an ETL process. Conversely, data stored in RDBMSs may need to be exported out or moved into a suitable format to perform complex data analysis. This movement of data adversely affects the time-to-insight. Recently a scalable system, called DaskDB, was introduced, which supports unified data analytics and in situ SQL query processing without requiring any data movement. It supports invoking existing Python API’s as User-Defined Functions (UDF) as a part of SQL queries, so they can be easily integrated with most of the existing Python applications. Due to the importance of supporting spatial analytics and spatial SQL queries, we have extended DaskDB to support spatial functionalities. In this paper, we present our enhanced DaskDB system. With two real-world spatial datasets, we demonstrate the scalability of DaskDB’s spatial features.