Evaluating regression test suites based on their fault exposure capability

SUMMARY The test process for evolving software systems takes on a different measurement aspect than that of new systems. Existing systems are generally being modified on a continuing basis as a normal part of the software maintenance activity. This process of product modifications is fault prone because faults are introduced in the code as it is being modified. From a statistical perspective, regression testing should be focused on those areas that are most likely to contain the introduced faults. Based on that premise, we have developed an evolutionary fault index that works as a fault surrogate varying in the same manner as faults. However, the knowledge as to the location of probable faults is not enough to assess the capabilities of a regression test suite. It is necessary to understand what the software is doing under each test. For that purpose, test execution profiles are gathered. Test execution profiles and the evolutionary fault indexes are combined in one methodology to provide an assessment of the overall regression testing activity and the suitability of each individual test. The methodology is illustrated with data from a 300 KLOC embedded system and its corresponding regression test suite. Copyright © 2000 John Wiley & Sons, Ltd.