Feature-oriented Test Case Selection during Evolution of Highly-Configurable Systems

Ensuring the quality of Highly Configurable Systems (HCSs) during its evolution and maintenance is challenging. As an HCS evolves, new features are added, changed, or removed, which makes the test case selection for regression testing a difficult task. The use of test traceability can help in this task, but there is a lack of studies exploring the use of trace links for HCS testing. Existing work is usually based on the variability model, which is not always available or updated. Yet, the few existing approaches rely on links between test cases and files/lines of code, limiting the selection to test cases related to file changes, not considering the whole implementation of features, which can be spread in many files other than the changed ones. Considering this limitation, this work presents a test case selection approach, namely FeaTestSel, that links test cases to features using HCS pre-processor directives. Then, the selection of test cases is based on features affected by changes in each commit. In addition to the selected test cases, the approach also produces the following reports to support the test activity: the lines of code that correspond to each feature, the lines exercised by each test case, and the test cases linked to each feature. To validate the approach, we rely on Libssh, a real open-source HCS in constant evolution. By adding the execution time of the approach to the execution time of the selected test cases, we achieved a reduction of approximately ≈50%, in comparison with the retest-all technique. Furthermore, the approach was able to maintain quality by selecting 100% of failed test files. The traceability and reports produced by our approach can also be used for further work by researchers, analysis of the test quality by engineers, or as a source of information for tool builders.