Into the ML-Universe: An improved classification and characterization of machine-learning projects

The prominence of Machine Learning (ML) systems led to the rise of Software Engineering for Artificial Intelligence (SE4AI), which addresses the unique engineering challenges of these systems. Researchers in SE4AI engage with three primary types of ML projects: those that apply ML techniques, those that develop new ML methodologies, and those that provide support tools and libraries. Current classification schemas distinguish ML projects based on their purpose and engineering quality, yet they miss a fine-grained classification of their nature and purpose. In this paper, we propose a novel, tool-supported automated classification schema for ML projects, coined Machine learning Automated Rule-based Classification Kit (MARK), that builds on top of the work by Gonzalez et al. to refine the classification of applied ML projects into ‘ML-Model Consumers,’ ‘ML-Model Producers,’ and ‘ML-Model Producers & Consumers.’ We evaluated MARK through two empirical studies. The first assessed its classification accuracy across 4,603 ML projects from two datasets. The second analyzed repository metrics, such as community engagement, activity, and structure, to demonstrate MARK’s potential in identifying trends and characteristics unique to each project type. Our findings indicate high F1-scores for our classifier, particularly for ‘ML-Model Producer’ projects, though challenges remain for ‘ML-Model Consumer’ classification. Significant differences in repository metrics among the classified projects highlight the usefulness of MARK, offering insights for researchers studying the socio-technical dynamics of ML projects.