CfExplainer: Explainable just-in-time defect prediction based on counterfactuals

Abstract

Just-in-time (JIT) defect prediction helps rationally allocate testing resources and reduce testing costs. However, most JIT defect prediction models lack explainability, which significantly affects their credibility. Recently, the local interpretable model-agnostic explanations (LIME) method has been used widely in model-explainable research, and many improved LIME-based methods have been proposed. However, problems with respect to explanation effectiveness and reliability remain, which seriously affects the practical use of LIME. To address this problem, CfExplainer, a local rule-based model-agnostic approach, is proposed. The approach first applies counterfactuals to generate synthetic instances. It then mines weighted class association rules based on synthetic instances, and it optimises the process of generating, ranking, pruning, and predicting the class association rules. Next, it employs the rules with the highest priority to explain the prediction results of the model. Experiments were conducted using the public datasets employed in related studies. Compared to other state-of-the-art methods, in terms of explanation effectiveness, CfExplainer's instance similarity improves by 26.5 %-31.2 %, and local model fittness improves by 2.0 %-3.5 %, 2.3 %-3 %, and 0.7 %-7.5 % on the AUC, F1-score, and Popt metrics, respectively. In terms of the reliability of the explanation, explanations that are 2.6 %-4.7 % more unique and 2.5 %-5.9 % more consistent with the actual characteristics of defect-introducing commits than other state-of-the-art methods. Thus, the explanations of the proposed approach can enhance the model credibility and help guide developers in fixing defects and reducing the risk of introducing them.