Feature-Interaction Sampling for Scenario-based Testing of Advanced Driver Assistance Systems✱

Abstract

Scenario-based testing is considered state-of-the-art to verify and validate Advanced Driver Assistance Systems. However, two essential unsolved challenges prevent the practical application of scenario-based testing according to the SOTIF-standard: (1) how to select a set of representative test scenarios, and (2) how to assess the effectiveness of a test scenario suite. In this paper, we leverage variability modelling techniques to select scenarios from a scenario space and assess the resulting scenario suites with a mutation score as metric. We capture the scenario space in a feature model and generate representative subsets with feature-interaction coverage sampling. The mutation score assesses the failure-finding effectiveness of these samples. We evaluate our concepts by sampling scenario suites for two independent Autonomous Emergency Braking function implementations and executing them on an industrial-strength simulator. Our results show that the feature model captures a scenario space that is relevant to identify all mutants. We show that sampling based on interaction coverage reduces the testing effort significantly while maintaining effectiveness in terms of mutation scores. Our results underline the potential of feature model sampling for testing in the automotive industry.