Optimizing Test Placement for Module-Level Regression Testing

Abstract

Modern build systems help increase developer productivityby performing incremental building and testing. Thesebuild systems view a software project as a group of interdependentmodules and perform regression test selection at themodule level. However, many large software projects have imprecisedependency graphs that lead to wasteful test executions. Ifa test belongs to a module that has more dependencies than theactual dependencies of the test, then it is executed unnecessarilywhenever a code change impacts those additional dependencies. In this paper, we formulate the problem of wasteful testexecutions due to suboptimal placement of tests in modules. We propose a greedy algorithm to reduce the number oftest executions by suggesting test movements while consideringhistorical build information and actual dependencies of tests. Wehave implemented our technique, called TestOptimizer, on topof CloudBuild, the build system developed within Microsoft overthe last few years. We have evaluated the technique on five largeproprietary projects. Our results show that the suggested testmovements can lead to a reduction of 21.66 million test executions(17.09%) across all our subject projects. We received encouragingfeedback from the developers of these projects, they accepted andintend to implement ≈80% of our reported suggestions.