Balancing Variability and Costs in Software Product Lines: An Experience Report in Safety-Critical Systems

This paper provides a detailed testing strategy for Software Product Lines (SPLs), aiming to balance the variability offered with its associated cost. Utilizing feature modeling, combinatorial testing, and deployment-based testing, the strategy addresses the unique challenges of testing SPLs, including the large number of possible feature interactions. The paper includes an experience report in the Syngo SPL by Siemens Healthineers that had approximately 900 optional features and was deployed at about 35.000 end-customer installations. Testing this variability using conventional approaches was an immense challenge due to the vast number of required test cases and test configurations. As a result, the actual variability offered to customers was restricted to four configurations, and the time-to-market of new features was limited to four releases per year. The project's goal was to devise a testing strategy to deliver a greater variety of configurations in a shorter time without substantially increasing testing effort or compromising quality. By restricting feature interaction testing to dependent features and those included in customer offerings, the strategy demonstrated the feasibility of maintaining high-quality deliverable configurations without significant increase in testing effort. Additionally, it allowed the accurate identification of necessary test case adjustments or developments when introducing new functionality, enabling significant reuse of existing test cases, and facilitating a transition to monthly release cycles. Overall, this paper provides valuable insights for practitioners and researchers working with SPLs and facing the challenges of testing them. The presented strategy offers a promising approach to reducing the number of cross-module test cases required in software platform development, demonstrating its feasibility and potential benefits.