Testing of User-Configurable Software Systems Using Firewalls

User-configurable software systems present many challenges to software testers. These systems are created to address a large number of possible uses, each of which is based on a specific configuration. As configurations are made up of groups of configurable elements and settings, a huge number of possible combinations exist. Since it is infeasible to test all configurations before release, many latent defects remain in the software once deployed. An incremental testing process is presented to address this problem, including examples of how it can be used with various user-configurable systems in the field. The proposed solution is evaluated with a set of empirical studies conducted on two separate ABB software systems using real customer configurations and changes. The three case studies analyzed failures reported by many different customers around the world and show that this incremental testing process is effective at detecting latent defects exposed by customer configuration changes in user-configurable systems.