Combining bug detection and test case generation

Abstract

Detecting bugs in software is an important software engineering activity. Static bug finding tools can assist in detecting bugs automatically, but they suffer from high false positive rates. Automatic test generation tools can generate test cases which can find bugs, but they suffer from an oracle problem. We present N-Prog, a hybrid of the two approaches. N-Prog iteratively presents the developer an interesting, real input/output pair. The developer either classifies it as a bug (when the output is incorrect) or adds it to the regression test suite (when the output is correct). N-Prog selects input/output pairs whose input produces different output on a mutated version of the program which passes the test suite of the original. In initial experiments, N-Prog detected bugs and rediscovered test cases that had been removed from a test suite.