Effect of test set size and block coverage on the fault detection effectiveness

Abstract

Size and code coverage are two important attributes that characterize a set of tests. When a program P is executed on elements of a test set T, we can observe the fault-detecting capacity of T for P. We can also observe the degree to which T induces code coverage on P according to some coverage criterion. We would like to know whether it is the size of T or the coverage of T on P which determines the fault detection effectiveness (FDE) of T for P. We found that there is little or no reduction in the FDE of a test set when its size is reduced while the all-uses coverage is kept constant. These data suggest, indirectly, that coverage is more correlated than the size with the FDE. To further investigate this suggestion, we report an empirical study to compare the statistical correlation between (1) FDE and coverage, and (2) FDE and the size. Results from our experiments indicate that the correlation between FDE and block coverage is higher than that between FDE and size.<>