Extracting Insights from Big Source Code Repositories with Automatic Clustering of Projects by File Names and Types

Abstract

Software project delivery requires a set of related activities to be conducted. The output of these activities form a collection of unstructured data such as specifications, requirements, manuals, source code and packaging files which is stored in configuration management systems. Software repositories are infrastructures to support project management activities and can be composed with several systems that include code change management, bug tracking, code review, build system, release binaries, wikis, forums, etc. This large and variable data collection provides opportunities for text mining tasks which further can be utilized for software delivery or business intelligence related goals. The proposed approach uses machine learning methods to inspect large software repositories and classifies the results to propose insights to project managers as an aide to make strategic business decisions. In the present work every software project is described by a certain representation (vector or set of vectors), which is constructed from names and types of the files from project content. These representations are used to find clusters of similar projects in big code repositories and highlight specific properties of the found groups: most used names and types of the file.