Improving Testing Efficiency using Cumulative Test Analysis

Abstract

It can be impossible to thoroughly test complex software projects with a large library of tests to be run in many environments and configurations. The cumulative test analysis (CTA) technique described reduces the time to find defects by prioritising and minimising the testing. Tests are chosen to target the product areas having the highest risk of defects. Test effectiveness, test code coverage, product code changes and changes to dependencies are monitored and analysed to prioritise the testing. Test results from build to build are accumulated. Build reports clearly identify areas at risk, test results, and the tests that must be run. Experiences with a prototype tool are discussed and conclusions drawn from the use of CTA show that defects are found sooner, more time is available for writing new tests and the focus of test execution moves towards product quality instead of simply test results