Hierarchical Metamorphic Relations for Testing Scientific Software

Abstract

Scientist developers have not yet routinely adopted systematic testing techniques to assure software quality. A key challenge is the oracle problem, a situation in which appropriate mechanisms are unavailable for checking if the code produces the expected output when executed using a set of test cases (TCs). Metamorphic testing alleviates the oracle problem by specifying the relationship that a source TC and its follow-up TC shall meet. Such relationships are called metamorphic relations (MRs) which are necessary properties of the intended program's functionality. Existing approaches handle the MRs in a flat manner. This paper introduces a novel way to facilitate a hierarchy of MRs to be developed incrementally. We illustrate our approach by testing U.S. EPA's Storm Water Management Model (SWMM). The results offer concrete insights into developing effective MRs to systematically test scientific software.