Empirically Comparing the Test Suite Reduction Techniques in Continuous Integration Process: How Test Suite Reduction Impacts Fault Localization

Abstract

In a continuous integration (CI) environment, software developers can frequently integrate the new or modified code to codebase and ensure the correctness of the modification by performing regression testing (RT). RT generally takes long time but the time period allocated for it is limited in practice. A CI environment can adopt test suite reduction (TSR) to improve RT and accelerate debugging by performing spectrum-based fault localization (SBFL). Yet, different TSR techniques will achieve different program test spectra and SBFL will operate based on these test spectra. Thus, the selection of TSR techniques has influence on both the effectiveness of the fault localization (FL) and the length of a CI period. If developers want to improve the CI by including TSR and SBFL, they have to choose a suitable TSR technique to maximize the FL effectiveness and minimize the CI period length. In this paper, we consider a TSR technique as the combination of the TSR strategy, the metric to evaluate test cases, and the coverage granularity level. We will analyze the impact caused by these three factors. Our study covers three TSR strategies, four metrics to evaluate test cases, and three coverage granularity levels, thus resulting in 36 TSR techniques. We investigate how each of the 36 TSR techniques affects the FL effectiveness and the CI period length by conducting the experiment based on seven subject programs that have been frequently adopted in the relevant studies. The experiment results indicate that performing TSR at branch coverage is recommended for optimizing the FL effectiveness while the HGS algorithm is suggested if the CI period length is developers’ main concern.