On strong mutation and subsuming mutants

Mutation analysis is a powerful technique for software testing but it is also known to be computationally expensive. The main reason for the high computational cost is that many of the mutants are redundant and thus, do not contribute to the quality of the test suite. One of the most promising approaches to avoid producing redundant mutants is to identify subsumption relations among mutants, preferably before these are generated. Such relations have for example, been identified at an operator level for mutants created by the ROR operator. This reduced set of non-redundant mutants has been used in several recent studies and is also the default option in at least one mutation testing tool that supports strong mutation. This raises questions on whether the identified subsumption relations between the mutants hold in a context of strong mutation or variants of weak mutation that require some limited error propagation (firm mutation). We have conducted an experimental study to investigate the subsumption relations in the context of strong or firm mutation. We observed that it is possible to create a test suite that is 100% adequate for the reduced set of mutants while not being 100% adequate for the complete set. This shows that the subsumption relations do not hold for strong or firm mutation. We provide several examples on this behavior and discuss the root causes. Our findings are important since strong and firm mutation both are frequently used to evaluate test suites and testing criteria. The choice of whether to use a reduced set of mutants or an entire set should however, not be made without consideration of the context in which they are used (i.e., strong, firm or weak mutation) since the subsumption relations between ROR mutants do not hold for strong or firm mutation. Just as redundant mutants can give an overestimation of the mutation score for a test suite, using the reduced set of mutants can give an underestimation if used together with strong or firm mutation. Results reported from such studies should therefore, be accompanied by information on whether the reduced or complete set of mutants was used and if the researchers used strong, firm or weak mutation.