Generating Controllably Invalid and Atypical Inputs for Robustness Testing

Abstract

One form of robustness in a software system is its ability to handle, in an appropriate manner, inputs that are unexpected compared to those it would experience in normal operation. In this paper we investigate a generic approach to generating such unexpected test inputs by extending a framework that we have previously developed for the automated creation of complex and high-structured test data. The approach is applied to the generation of valid inputs that are atypical as well as inputs that are invalid. We demonstrate that our approach enables control of the 'degree' to which the test data is invalid or atypical, and show empirically that this can alter the extent to which the robustness of a software system is exercised during testing.