Software Defect Prediction evaluation: New metrics based on the ROC curve

Abstract

Context:

ROC (Receiver Operating Characteristic) curves are widely used to represent how well fault-proneness models (e.g., probability models) classify software modules as faulty or non-faulty. AUC, the Area Under the ROC Curve, is usually used to quantify the overall discriminating power of a fault-proneness model. Alternative indicators proposed, e.g., RRA (Ratio of Relevant Areas), consider the area under a portion of a ROC curve. Each point of a ROC curve represents a binary classifier, obtained by setting a specified threshold on the fault-proneness model. Several performance metrics (Precision, Recall, the F-score, etc.) are used to assess a binary classifier.

Objectives:

We investigate the relationships linking “under the ROC curve area” indicators such as AUC and RRA to performance metrics.

Methods:

We study these relationships analytically. We introduce iso-PM ROC curves, whose points have the same value $\overline{PM}$ for a given performance metric PM. When evaluating a ROC curve, we identify the iso-PM curve with the same value of AUC or RRA. Its $\overline{PM}$ can be seen as a property of the ROC curve and fault-proneness model under evaluation.

Results:

There is an S-shaped relationship between $\overline{PM}$ and AUC for performance metrics that do not depend on the proportion $\rho$ of faulty modules, i.e., dataset balancedness. $\varphi$ (Matthews Correlation Coefficient) depends on $\rho$: with very imbalanced datasets, AUC appears over-optimistic and $\varphi$ over-pessimistic. RRA defines the region of interest in terms of $\rho$, so all performance metrics depend on $\rho$. RRA is related to performance metrics via S-shaped curves.

Conclusion:

Our proposal helps gain a better quantitative understanding of the goodness of a ROC curve, especially in practically relevant regions of interest. Also, showing a ROC curve and iso-PM curves provides an intuitive perception of the goodness of a fault-proneness model.