How Well are Your Requirements Tested?

Abstract

We address the question: to what extent does covering requirements ensure that a test suite is effective at revealing faults? To answer it, we generate minimal test suites that coverall requirements, and assess the tests they contain. They turn out to be very poor -- ultimately because the notion of covering a requirement is more subtle than it appears to be at first. We propose several improvements to requirements tracking during testing, which enable us to generate minimal test suites close to what a human developer would write. However, there remains a class of plausible bugs which such suites are very poor at finding, but which random testing finds rather easily.