Testing Scientific Software with Invariant Relations: A Case Study

Adequately testing scientific software is essential to the quality of the software. However, it is a grand challenge due to the oracle problem. Metamorphic testing has shown its effectiveness for alleviating the problem. But the effectiveness of metamorphic testing is highly dependent on the quality of metamorphic relations that are developed for testing the software. In this paper, we propose a framework for iteratively developing metamorphic relations for adequately testing scientific software. The basic idea is to refine metamorphic relations that are loosely defined to those that can be verified with only limited number of cases so that the relations can be accurately tested. We explain the framework through testing a scientific software system that is used for modeling light scattering of particles. Based on domain knowledge and general guidelines, a group of metamorphic relations are first identified and tested. According to testing results, the metamorphic relations are refined step by step until each of them can be accurately tested. In particular, an invariant transform is applied to the metamorphic relations for significantly reducing the number of cases that can satisfy the relations. Finally the relations are further transformed with a carefully defined hash function to ensure each of the metamorphic relations can be automatically verified. The proposed approach truly solves the oracle problem and greatly improves the effectiveness of metamorphic testing. Its effectiveness is evaluated by mutation testing and demonstrated by new problems found in the software.