An Empirical Study of the Relationship between Continuous Integration and Test Code Evolution

Abstract

Continuous Integration (CI) is the practice of automating and improving the frequency of code integration. CI has been widely adopted by software development teams and has brought the attention of researchers to study its benefits. Existing research shows that CI can improve software quality by identifying the errors earlier in the software development life-cycle. One question that remains open, however, is whether CI increases the adoption of testing practices in software projects. The goal of our work is to investigate the evolution of software tests and its relationship with the adoption of Continuous Integration. We set out to compare 82 projects that adopted CI (CI projects) and 82 projects that have never adopted CI (NOCI projects). In total, we studied 3,936 versions of our studied projects to investigate trends on the test code ratio and coverage. We observe that 40.2% of the CI projects have a rising test-code ratio trend while only 17% of the NOCI projects have a rising trend. Additionally, we find evidences that CI projects have improved the overall test coverage while NOCI projects do not experience the same growth. Finally, we build a mixed-effects model to study software development factors than can possibly explain the test ratio. Our models reveal that test ratio is largely explained by the project inherent context rather than code or process factors. In overall, our work demonstrates that Continuous Integration can be empirically associated with a healthier test code evolution.