Test Effectiveness Evaluation of Prioritized Combinatorial Testing: A Case Study

Combinatorial testing is a widely-used technique to detect system interaction failures. To improve test effectiveness with given priority weights of parameter values in a system under test, prioritized combinatorial testing constructs test suites where highly weighted parameter values appear earlier or more frequently. Such order-focused and frequency-focused combinatorial test generation algorithms have been evaluated using metrics called weight coverage and KL divergence but not sufficiently with fault detection effectiveness so far. We evaluate the fault detection effectiveness on a collection of open source utilities, applying prioritized combinatorial test generation and investigating its correlation with weight coverage and KL divergence.