Ensuring Reliability Through Combinatorial Coverage Measures

Verification of complex software systems is an important, yet challenging task. Testing is the most common method for assuring that software meets its specifications and is defect-free. To claim that software is defect-free and thus reliable, one has to show that it produces the “correct” output or “behaves” according to specification without failing under all possible parameter values and configurations. In the software verification world, this is known as exhaustive testing. For any software of reasonable size and complexity, exhaustive testing is completely infeasible. Thus, during the testing process, a small subset of parameter values and configurations is selected to ensure that the software is producing its output or maintaining its behavior as “expected.” The selected parameter value for one test execution is called a Test Case, and the set of test cases selected for testing a system is called a Test Suite. The essence of software testing, therefore, lies in effective ways of identifying the test cases and building the test suite. Two overarching questions related to this process include: 1) how to select the test cases and 2) how to decide when enough test cases have been selected. Over the years, researchers have proposed different Test Adequacy criteria for answering these two questions.