An Empirical Comparison of Similarity Measures for Abstract Test Case Prioritization

Abstract

Test case prioritization (TCP) attempts to order test cases such that those which are more important, according to some criterion or measurement, are executed earlier. TCP has been applied in many testing situations, including, for example, regression testing. An abstract test case (also called a model input) is an important type of test case, and has been widely used in practice, such as in configurable systems and software product lines. Similarity-based test case prioritization (STCP) has been proven to be cost-effective for abstract test cases (ATCs), but because there are many similarity measures which could be used to evaluate ATCs and to support STCP, we face the following question: How can we choose the similarity measure(s) for prioritizing ATCs that will deliver the most effective results? To address this, we studied fourteen measures and two popular STCP algorithms — local STCP (LSTCP), and global STCP (GSTCP). We also conducted an empirical study of five realworld programs, and investigated the efficacy of each similarity measure, according to the interaction coverage rate and fault detection rate. The results of these studies show that GSTCP outperforms LSTCP — in 61% to 84% of the cases, in terms of interaction coverage rates; and in 76% to 78% of the cases with respect to fault detection rates. Our studies also show that Overlap, the simplest similarity measure examined in this study, could obtain the overall best performance for LSTCP; and that Goodall3 has the best performance for GSTCP.