Parallelization in System-Level Testing: Novel Approaches to Manage Test Suite Dependencies

Abstract

System-level testing is fundamental to ensure the reliability of software systems. However, the execution time for system tests can be quite long, sometimes prohibitively long, especially in a regimen of continuous integration and deployment. One way to speed things up is to run the tests in parallel, provided that the execution schedule respects any dependency between tests. We present two novel approaches to detect dependencies in system-level tests, namely Pfast and Mem-Fast, which are highly parallelizable and optimistically run test schedules to exclude many dependencies when there are no failures. We evaluated our approaches both asymptotically and practically, on six Web applications and their system-level test suites, as well as on MySQL system-level tests. Our results show that, in general, Pfast is significantly faster than the state-of-the-art PraDet dependency detection algorithm, while producing parallelizable schedules that achieve a significant reduction in the overall test suite execution time.