Exploiting data lineage for parallel optimization in extensible DBMSs

Abstract

Extensibility and high query performance are important requirements of advanced large scale information systems since complex data analysis often requires the use of application-specific operations that have to be introduced by the user issuing the query. Towards the goal of supporting automatic parallelization of queries containing complex user-defined evaluators in an extensible DBMS, we devised a relevance window model to capture the inherent data lineage characteristics of evaluators on multidimensional data sets. Informally, the relevance window of an evaluator defines the scope of influence input data records have on the value of records in the output data space. An evaluator's relevance window constrains the data partitioning opportunities available for an evaluator.