Top-down: A better strategy for incremental covering array generation

Abstract

Context:

The Incremental Covering Array (ICA) offers a flexible and efficient test schedule for Combinatorial Testing (CT) by enabling dynamic adjustment of test strength. Despite its importance, ICA generation has been under-explored in the CT community, resulting in limited and suboptimal existing approaches.

Objective:

To address this gap, we introduce a novel strategy, namely Top-down, for optimizing ICA generation.

Method:

In contrast to the traditional strategy, named Bottom-up, Top-down starts with a higher-strength test set and then extracts lower-strength sets from it, thus leveraging test case generation algorithms more effectively.

Results:

We conducted a comparative evaluation of the two strategies across 17 real-world software with 84 total versions. The results demonstrate that compared with Bottom-up, the Top-down strategy requires less time and generates smaller ICAs while covering more higher-strength interactions. Furthermore, Top-down outperforms Bottom-up in early fault detection and code line coverage, while also surpassing the random and direct CA generation strategies.

Conclusion:

The Top-down strategy not only improved the efficiency of test case generation but also enhanced the effectiveness of fault detection in the incremental testing scenarios.