Three trillion lines: infrastructure for mining GitHub in the classroom

Abstract

The increasing interest in collaborative software development on platforms like GitHub has led to the availability of large amounts of data about development activities. The GHTorrent project has recorded a significant proportion of GitHub’s public event stream and hosts the currently largest public dataset of meta-data about open-source development. We describe our infrastructure that makes this data locally available to researchers and students, examples for research activities carried out on this infrastructure, and what we learned from building the system. We identify a need for domain-specific tools, especially databases, that can deal with large-scale code repositories and associated meta-data and outline open challenges to use them more effectively for research and machine learning settings.