My feature model has changed... What should I do with my tests?

Abstract

Software Product Lines (SPLs) evolve over time, driven by changing requirements and advancements in technology. While much research has been dedicated to the evolution of feature models (FMs), less focus has been put on how associated artifacts, such as test cases, should adapt to these changes. Test cases, derived as valid products from an FM, play a critical role in ensuring the correctness of an SPL. However, when an FM evolves, the original test suite may become outdated, requiring either regeneration from scratch or repair of existing test cases to align with the updated FM. In this paper, we address the challenge of evolving test suites upon FM evolution. We introduce novel definitions of test suite dissimilarity and specificity We use these metrics to evaluate three test generation strategies: GFS (generating a new suite from scratch), GFE (repairing and reusing an existing suite), and SPECGEN (maximizing specific tests for the FM evolution). Additionally, we introduce a set of mutations to simulate FM evolution and obtain additional FMs. By using mutants, we conduct our analyses and evaluate the mutation score of test generation strategies. Our experiments, conducted on a set of FMs taken from the literature and on more than 3,200 FMs artificially generated with mutations, reveal that GFE often produces the smallest test suites with high mutation scores, while SPECGEN excels in specificity, particularly for mutations expanding the set of valid products.