Assessing the strength of Metamorphic Testing applied to optimisation software—Experience from industry

Abstract

Context:

The testing of optimisation algorithms (OAs) is difficult due to the test oracle problem. Metamorphic Testing (MT) addresses this challenge. We previously applied MT to a black-box industrial OA and gained first insights into applying MT for OA testing.

Objective:

We noticed that some of the identified Metamorphic Relations (MRs) seemed to be more powerful than others. We now define and evaluate an approach to assess and rank MRs in terms of their defect-detection capabilities.

Method:

We propose a three-phase approach for assessing the strength of MRs. First, we evaluate the applicability of each MR based on Test Data (TD) using MetaTrimmer, an approach for selecting and constraining MRs. Second, we generate System Under Test (SUT) mutants and classify them into three levels. Level one contains mutants that do not change their behaviour for all TD (equivalent mutants). Level two contains all mutants that change behaviour for every TD (trivial mutants). Level three contains all other mutants. Third, we assess MR effectiveness per level based on the MR’s violation/non-violation ratio.

Results:

Among 405 generated SUT mutants analysed, 236 fell into level one, 85 into level two, and 84 into level three. The analysis of the amount of TD triggering an MR violation in each level per mutant revealed that some MRs have higher sensitivity than others.

Conclusion:

Our findings show that assessing MRs using our three-level strategy provides clear insights into their defect-detection capabilities. By integrating MetaTrimmer with mutation testing, we identify MRs that are effective in catching faults and sensitive to variations in TD. While evaluated on an OA in an industrial setting, this approach is generalisable to other SUTs. It is actionable for practitioners and researchers seeking to identify robust MRs and offers a structured methodology to evaluate and rank them.