SmokedDuck Demonstration: SQLStepper

Abstract

Fine-grained lineage tracks the relationships between input and output of a query, and is particularly useful in analytical applications such as query debugging, view maintenance, query explanations, and data cleaning. Prior approaches rewrite SQL queries to also track lineage, but can slow query execution in analytical engines that are designed to process complex query patterns on large datasets. Moreover, they mainly capture lineage at the logical level. SmokedDuck extends DuckDB to support fast lineage capture and querying by tracking lineage at the instruction level by leveraging the duality between lineage and data movement. In this demonstration, we show how a user can leverage operator-level lineage to understand and debug a query execution through SQLStepper: an application built on top of SmokedDuck. Users upload data and execute queries using an in-browser command line, then explore query-level and operator-level lineage visually to track down bugs.