Interactive refinement of combinatorial test plans

Abstract

Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interactions in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. When applying CTD to real-life testing problems, it can often occur that the result of CTD cannot be used as is, and manual modifications to the tests are performed. One example is very limited resources that significantly reduce the number of tests that can be used. Another example is complex restrictions that are not captured in the model of the test space. The main concern is that manually modifying the result of CTD might potentially introduce coverage gaps that the user is unaware of. In this paper we present a tool that supports interactive modification of a combinatorial test plan, both manually and with tool assistance. For each modification, the tool displays the new coverage gaps that will be introduced, and enables the user to take educated decisions on what to include in the final set of tests.