Using Metamorphic Relation Violation Regions to Support a Simulation Framework for the Process of Metamorphic Testing

Abstract

Metamorphic testing (MT) has been growing in pop-ularity, but it can still be quite challenging and time-consuming to assess its performance. Typical approaches to performance assessment can require a series of steps, and depend on a variety of factors, often requiring serendipity. This can be a bottleneck for some aspects of MT research. Central to MT, metamorphic relations (MRs) represent necessary properties of the system under test (SUT). In traditional software testing, simulations are often employed to examine and compare the performance of dif-ferent testing strategies. However, these simulations are typically designed based on the assumed availability (and applicability) of a test oracle - a mechanism to decide the correctness of the SUT output or behaviour. A key reason for the popularity of MT is its proven record of effective software testing, without the need for a test oracle. This strength, however, also means that traditional ways of using simulations to analyse software testing approaches are not applicable for MT. This lack of cheap and fast ways to conduct simulation analyses of MT is a hurdle for many aspects of MT research, and may be an obstacle to its more widespread adoption. To address this, in this paper we introduce the concept of MR-violation regions (MRVRs), and show how they can be used for a certain category of MRs, Deterministic MRs (DMRs), to build simulation tools for MT. We analyse the differences between MRVRs and traditional, oracle-defined failure regions; and report on a preliminary case study exploring MRVRs in numerical-input-domain systems from previous MT studies. We anticipate that the proposed MT simulation framework may facilitate more research into MT, and may help lead to its more widespread adoption.