Fault model-based variability testing

Abstract

Software Product Lines (SPL) testing techniques are commonly focused on handling variability from a high level abstraction perspective, despite the importance of understanding the nature of issues emerging from source code that could affect the overall quality of products. In this investigation, we present a framework aimed to handle such a neglected issue by augmenting an SPL testing process with fault modeling support. Fault modeling is an strategy employed to capture the behaviour of the system against faults. By understanding the nature of faults before developing the tests might improve the likelihood of finding particular classes of errors. The proposed framework encompasses test assessment, to evaluate the effectiveness of existing test suites, and test design, by focusing on fault-prone elements. We carried out a controlled experiment to assess the test effectiveness of the proposed framework. Software engineers from an industrial partner acted as subjects. The assessment has shown promising results that confirm the hypothesis that combining fault models in an SPL testing process performs significantly better on improving the quality of test inputs.