Combinatorial transition testing in dynamically adaptive systems: Implementation and test oracle

Due to the large number of possible interactions and transitions among features in dynamically adaptive systems, testing such systems poses significant challenges. To verify that such systems behave correctly, combinatorial interaction testing (CIT) can create concise test suites covering all valid pairs of features of such systems. While CIT claims to find all errors caused by two features, it does not cover certain errors occurring only for specific transitions between features. To address this issue we study the technique of Combinatorial Transition Testing (CTT), which includes both generation and detection of what we call behavioural transition errors. From an initial generation algorithm that combines both interaction and transition coverage but lacks scalability, we propose an optimised version that enables CTT even for hundreds of features. From a valid test suite covering all transitions, we complete our testing approach with a test oracle that detects all behavioural transition errors without any prior knowledge of the system’s behaviour. After a comprehensive analysis over a large number of feature models, we conclude that size of CTT-generated test suites and test effort needed to use our test oracle are linearly correlated to CIT-generated ones and that CTT grows logarithmically in the number of features.