Generating Pairwise Covering Arrays for Highly Configurable Software Systems

Abstract

Highly configurable software systems play crucial roles in real-world applications, which urgently calls for useful testing methods. Combinatorial interaction testing (CIT) is an effective methodology for detecting those faults that are triggered by the interaction of any t options, where t is the testing strength. Pairwise testing, i.e., CIT with t = 2, is known to be the most practical and popular CIT technique, and the pairwise covering array generation (PCAG) problem is the most critical problem in pairwise testing. Due to the practical importance of PCAG, many PCAG algorithms have been proposed. Unfortunately, existing PCAG algorithms suffer from the severe scalability problem. To this end, the SPLC Scalability Challenge (i.e., Product Sampling for Product Lines: The Scalability Challenge) has been proposed since 2019, in order to motivate researchers to develop practical PCAG algorithms for overcoming this scalability problem. In this work, we present a practical PCAG algorithm dubbed SamplingCA-ASF. To the best of our knowledge, our experiments show that SamplingCA-ASF is the first algorithm that can generate PCAs for Automotive02 and Linux, the two hardest and largest-scale instances in the SPLC Scalability Challenge, within reasonable time. Our experimental results indicate that SamplingCA-ASF can effectively alleviate the scalability problem in pairwise testing.