Harnessing Multiple Source Test Cases in Metamorphic Testing: A Case Study in Bioinformatics

Abstract

Metamorphic testing (MT) has been applied to software verification, validation and quality assessment. In mostprevious studies, research has focused on deriving metamorphic relations (MRs) such that the input of one or more follow-up testcases is generated from one source test case. We note that some programs under test (PUT) naturally take multiple inputs andprocess them simultaneously to generate multiple outputs. This type of programs are common in the field of big data analysisand bioinformatics. This means, in the source execution of the program, we can obtain multiple outputs from multiple sourcetest cases. Here we consider a type of MR in which multiple follow-up test cases are generated from multiple source testcases simultaneously. We hypothesise that harnessing the outputs from multiple source test cases enables us to obtain additionalinformation about the PUT, and therefore allows us to construct more effective MRs. In this paper, we designed a new MR to testa popular RNA sequence alignment program. Since the MR was designed based on a desirable property of the PUT (which usesa complex heuristic algorithm) rather than a necessary property, violation of this MR indicates the program outputs deviate fromuser expectation, hence this MR can be used for assessing the quality of the outputs. Furthermore, we note that outputs ofthe follow-up test cases allow us to putatively assign alignment information to some sequences that were not aligned in the sourcetest cases, hence potentially identifying and correcting these 'false negative' outputs. We believe this case study provides importantinsight into designing MRs based on multiple source test cases, and how testing results can be used to improve the performanceof some heuristic-based programs.