Method-Level Test Selection for Continuous Integration with Static Dependencies and Dynamic Execution Rules

In Continuous Integration (CI) development environment, integration testing (i.e., CI testing) is an important practice to verify the quality of submitted code. With the growth of integration system, running all tests leads to high test cost with slow feedback. Many test case selection techniques have been proposed to tackle this problem. We analyze existing static test selection approaches on method-level, and find there are two main drawbacks, i.e., omission of dependencies and imprecise dependencies, which influence the effectiveness of test selection. Both of them are due to the fact that some dependencies are dynamically determined at runtime, which could not be resolved solely by current static analysis techniques. We propose a Method-level tEst SelecTion approach (i.e., MEST), which utilizes static dependencies and dynamic execution rules (i.e., dynamic invocation in reflection and dynamic binding in inheritance). Evaluation is conducted on 18 open source projects with 261 continuous integration versions from Eclipse and Apache communities. We assess the effectiveness of MEST from reduced test size, fault detection efficiency and test cost, and compare it with the state-of-the-art approach ClassSRTS (as baseline); and further analyze the contribution of dynamic execution rules. Results show that (1) on average, MEST can reduce test size by 92% and 43% compared with actual CI testing and baseline respectively; (2) MEST can fully cover all faults detected by actual CI testing (in 97% versions) and baseline (in 98% versions), and find new faults in 26% and 27% versions respectively; (3) on average, the endto-end time of MEST is 24% and 48% of actual CI testing and baseline respectively; (4) both dynamic execution rules contribute to fault detection through capturing necessary dependencies. This approach can further speed up the feedback of CI testing and improve the fault detection efficiency of CI testing.