Continuous T-Wise Coverage

Abstract

Quality assurance for highly configurable systems uses t-wise feature interaction coverage as a metric to measure the quality of selected samples for testing. Achieving t-wise feature interaction coverage requires testing many configurations, often exceeding the available testing time for frequently evolving systems. As testing time is a limiting factor, current testing procedures face the challenge of finding a reasonable trade-off between achieving t-wise feature interaction coverage and reducing the time required for testing. To address this challenge, we can consider t-wise feature interactions covered in previous test executions when calculating the achieved t-wise feature interaction coverage. However, the current definition of t-wise feature interaction coverage does not consider previously tested configurations. Therefore, we propose continuous t-wise coverage as a new customizable metric for tracking the ratio of achieved t-wise feature interaction coverage over time. Our metric allows customizing the tradeoff between test effort per system version and the time to achieve t-wise coverage. We evaluate various parameterizations for our metric on four real-world evolution histories and investigate how they impact the calculated t-wise feature interaction coverage. Our results show that a high t-wise feature interaction coverage can be achieved by testing significant (up to 50%) smaller samples per commit, when the evolution of the system is considered.