An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique

Abstract

Compilers play an important role in software development, and it is quite necessary to perform abundant testing to ensure the correctness of compilers. A critical task in compiler testing is to validate the semantic-soundness property which requires consistence between semantics of source programs and behavior of target executables. For validating this property, one main challenging issue is generation of a test oracle. Most existing approaches fall into two main categories when dealing with this issue: reference-based approaches and assertion-based approaches. All these approaches have their weakness when new programming languages are involved or test automation is required. To overcome the weakness in the existing approaches, we propose a new automatic approach for testing compiler. Our approach is based on the technique of metamorphic testing, which validates software systems via so-called "metamorphic relations". We select the equivalence-preservation relation as the metamorphic relation and propose an automatic metamorphic testing framework for compiler. We also propose three different techniques for automatically generating equivalent source programs as test inputs. Based on our approach, we developed a tool called Mettoc. Our mutation experiments show that Mettoc is effective to reveal compilers' errors in terms of the semantic-soundness property. Moreover, the empirical results also reveal that simple approaches for constructing test inputs are not weaker than complicated ones in terms of fault-detection capability. We also applied Mettoc in testing a number of open source compilers, and two real errors in GCC-4.4.3 and UCC-1.6 respectively have been detected by Mettoc.