Efficient Query Processing in Python Using Compilation

Abstract

In this paper, we present a framework for efficient query processing in Python. Inspired by the increasing interest in Python-based frameworks such as TensorFlow and Pandas for data scientists, our framework consists of three different input languages. The first language is SQL; to better integrate the SQL queries with the rest of the data science pipeline, by relying on off-the-shelf query optimizers (e.g., PostgreSQL) the SQL code is translated to a physical query plan, which is in turn translated to Pandas code. The second input is Pandas code; it can be either run by Pandas itself or alternatively be translated into SDQL.py, the third input language that can be translated into efficient low-level code and can achieve an order-of-magnitude performance improvement over Pandas. Our framework exposes a Python-based API that allows data scientists to use SDQL.py as a pure Python library.