Are We There Yet? How Redundant and Equivalent Mutants Affect Determination of Test Completeness

Mutation score has long been used in research as a metric to measure the effectiveness of testing strategies. This paper presents evidence that mutation score lacks the desired accuracy to determine the completeness of a test suite due to noise introduced by the redundancy inherent in traditional mutation, and that dominator mutation score is a superior metric for this purpose. We evaluate the impact of different levels of redundant and equivalent mutants on mutation score and the ability to determine completeness in developing a mutation-adequate test suite. We conclude that, in the context of our model, redundant mutants make it very difficult to accurately assess test completeness. Equivalent mutants, on the other hand, have little effect on determining completeness. Based on this information, we suggest limits to redundancy and equivalency that mutation tools must achieve to be practical for general use in software testing.