How Well Just-In-Time Defect Prediction Techniques Enhance Software Reliability?

Abstract

Many Just-In-Time defect prediction (JIT) techniques, which anticipate defect-prone software changes, have been proposed in recent years. Researchers have evaluated these techniques from different perspectives and have drawn inconsistent conclusions about which JIT defect prediction techniques are the most effective and efficient. This paper evaluates JIT techniques from a reliability perspective. For short-term early evaluation, we measure JIT predictive performance on early exposed defects. While for long-term evaluation, we quantify the overall reliability improvement resulted from JIT. A case study applying 11 state-of-the-art JIT methods on 18 large open-source projects has shown: 1) Different JIT methods have their own individual strengths for different purposes, 2) in general, RandomForest is the most effective method in short-term software reliability improvement, and CBS+ performs best in long-term reliability improvement; 3) JIT prediction accuracy is highly correlated to overall reliability improvement.