An Empirical Study on the Cross-Project Predictability of Continuous Integration Outcomes

Build prediction can reduce latency between continuous integration outcomes and the corresponding decisions, improving the efficiency of development team. Current build predictions are generally within-project, making it unavailable on those projects without enough build data. Cross-project prediction is the-state-of-art technique to solve the lack of training data on the studied projects by importing data from other projects. However, no previous study focuses on cross-project build predictions and checks the performance in the real world projects. This paper carries out an empirical study on the performance of cross-project build prediction with a wide range of 126 opensource projects under 6 common classifiers. In this paper, to select the training sets for cross-project prediction, we introduce two widely used data selection methods: Burak Filter based on build-level and Bellwether Strategy based on project-level. According to the results of our experiments, we have the following observations. Firstly, by the comparison between these two methods, we find that project-level selection (Bellwether strategy) performs better than build-level selection (Burak Filter). Furthermore, we observe that the prediction results can be improved by clustering the 126 studied projects into several smaller communities containing about 20-40 projects. And among 6 used classifiers, we find that decision tree classifier performs the best. Finally, by computing the optimal prediction results, we conclude that current selection methods still need to be improved to get close to the optimal prediction in cross-project build predictions.