Using Metamorphic Testing to Improve Dynamic Symbolic Execution

Abstract

Dynamic symbolic execution (DSE) is an approach for automatically generating test inputs from source code using constraint information. It is used in fuzzing: the execution of tests while monitoring for generic properties such as buffer overflows and other security violations. Limitations of DSE for fuzzing are two-fold: (1) only generic properties are checked: many deviations from specified behaviour are not found, and (2) many programs are not entirely amenable to DSE because they give rise to hard constraints, so that some parts of a program remain uncovered. In this paper, we discuss how to mitigate these problems using metamorphic testing (MT). Metamorphic testing uses domain-specific properties about program behaviour, relating pairs of inputs to pairs of outputs. From a given test suite, follow-up tests inputs are generated, and their outputs are compared to outputs from the original tests, using metamorphic relations. Our hypothesis is that using metamorphic testing increases the ability of a DSE test suite to find faults, and that the follow-up tests execute some previously-uncovered segments. We have experimented with seven small but non-trivial libraries, comparing DSE test suites with DSE+MT test suites, demonstrating that DSE+MT test suites improve coverage marginally, but find more faults.