On the Non-Generalizability in Bug Prediction

Abstract

Bug prediction is a technique used to estimate the most bug-prone entities in software systems. Bug prediction approaches vary in many design options, such as dependent variables, independent variables, and machine learning models. Choosing the right combination of design options to build an effective bug predictor is hard. Previous studies do not consider this complexity and draw conclusions based on fewer-than-necessary experiments. We argue that each software project is unique from the perspective of its development process. Consequently, metrics and machine learning models perform differently on different projects, in the context of bug prediction. We confirm our hypothesis empirically by running different bug predictors on different systems. We show there are no universal bug prediction configurations that work on all projects.