Bow tie testing: a testing pattern for product lines

Abstract

Verification of highly configurable systems poses a significant challenge, the challenge of knowing that every configuration works when there often are intractably many different configurations. When a homogeneous abstraction layer has many mutually exclusive alternative implementations, we might, according to the polymorphic server test pattern, test these implementations using one test suite targeted towards the abstraction layer which is then run for each concrete implementation of the abstraction layer. But, the pattern does not handle interaction testing. Combinatorial interaction testing is one of the more promising techniques for doing interaction testing of a software product line. The bow tie testing pattern describes how the configurations which differ only in the implementation layer require one test suite. In addition, comparing the execution results of one product with another provides for a test oracle. The pattern reduces the effort of testing a highly configurable system without reducing the error detection capabilities provided by ordinary combinatorial interaction testing. We present an example of a subset of the Eclipse IDE product line, and show that only 20 test suites is required to test 41 products, a significant reduction.