Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites

Abstract

In this paper we assess and evaluate the quality of t-way combinatorial test-suites using three different test-suite quality assessment methods. As t-way combinatorial test-suites reduce the input space of a program under test, we investigate how an increasing t affects the quality of the test-suite. There are some limitations of existing test-suite quality assessment methods e.g. the number of mutants is limited by execution time and code coverage measurement might be intrusive due to changes of the behavior of the program under test when instrumenting the code. Here we generate t-way combinatorial test-suites for Java programs of different size. We compute mutation score and code coverage for the generated test-suites, and apply additionally a new model inference based approach, that does not require to execute the program under test, to compare the generated test-suites with each other and assign a quality valuation to the test-suites. Our results show that an increasing t generally raises test-suite quality in terms of mutation score, coverage, and model inference. However, the model inference approach is only applicable, if the outcomes of the programs under test are discrete values, and if the number of discrete values is less than the test-suite size.