Testing Tools for Configurable Software Systems: A Review-based Empirical Study

Configurable software systems are software systems that can be adapted or configured according to a set of features with the goal of increasing reuse and productivity. However, testing configurable systems is very challenging due to the number of configurations to run with each test, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing techniques and tools have been proposed to deal with this challenge, but their potential practical application remains mostly unexplored. The lack of studies to explore the tools that apply those techniques motivated us to investigate the literature to find testing tools for configurable software systems and to understand how they work. In this paper, we conducted a systematic mapping and identified 34 testing tools for configurable software systems. We first summarized and discussed their main characteristics. We then designed and performed a comparative empirical study of the main sound testing tools found: VarexJ and SPLat. They are considered sound testing techniques because they explore all reachable configurations from a given test. Overall, we observed that VarexJ and SPLat presented distinct results for efficiency while testing the target systems and that, although VarexJ found more errors than SPLat for the majority of the target systems, such result deserves a more in-depth investigation because we expected a higher intersection of errors encountered by them.