GitSchemas: A Dataset for Automating Relational Data Preparation Tasks

Abstract

The preparation of relational data for machine learning (ML) has largely remained a manual, labor-intensive process, while automated machine learning has made great strides in recent years. Long-standing challenges, such as reliable foreign key detection still pose a major hurdle towards more automation of data integration and preparation tasks. We created a new dataset aimed at increasing the level of automation of data preparation tasks for relational data. The dataset, called GITSCHEMAS, consists of schema metadata for almost 50k real-world databases, collected from public GitHub repositories. To our knowledge, this is the largest dataset of such kind, containing approximately 300k table names, 2M column names including data types, and 100k real (not semantically inferred) foreign key relationships. In this paper, we describe how Gitschemaswas created, and provide key insights into the dataset. Furthermore, we show how GITSCHEMAS can be used to find relevant tables for data augmentation in an AutoML setting.