Unifying data exploration and curation

Abstract

Recent years have seen a surge in "self-service" business intelligence tools. These tools primarily focus on supporting decision-making by non-technical "end users", through data exploration -- the querying of data and inspection of results. Exploration, however, is only part of the story. Curation is its complement. Curation is the ability to organize data into structures that are meaningful for a particular problem domain and convenient for building further explorations upon. Curation is also the ability to modify data, as well as creating new data through rules and constraints, in order to support what-if's, forecasting, and planning for the future. Exploration and curation often need to interleave in the decision-making process of an end-user. In this talk, we discuss the LogicBlox Modeler, a unifying environment that provides support for both exploration and curation. We motivate the need for a unifying environment through applications in government, major financial institutions, and large global retailers. We discuss our language -- in its visual and textual representations -- that supports not only querying, but also the creation and modification of schema and data. We discuss the challenges imposed on the database runtime by the use cases of exploration and curation at scale and aspects of the LogicBlox database designed to meet these challenges.