Statistical Metamorphic Testing Testing Programs with Random Output by Means of Statistical Hypothesis Tests and Metamorphic Testing

Abstract

Testing software with random output is a challenging task as the output corresponding to a given input differs from execution to execution. Therefore, the usual approaches to software testing are not applicable to randomized software. Instead, statistical hypothesis tests have been proposed for testing those applications. To apply these statistical hypothesis tests, either knowledge about the theoretical values of statistical characteristics of the program output (e. g. the mean) or a reference implementation (e. g. a legacy system) are required to apply statistical hypothesis tests. But often, both are not available. In the present paper, it is discussed how a testing method called Metamorphic Testing can be used to construct statistical hypothesis tests without knowing exact theoretical characteristics or having a reference implementation. For that purpose, two or more independent output sequences are generated by the implementation under test (IUT). Then, these sequences are compared according to the metamorphic relation using statistical hypothesis tests.