Improving scenario testing process by adding value-based prioritization: an industrial case study

Most of the current testing strategies treat all aspects of software equally important in a value-neutral way; this becomes more risky when the testing resources are limited. Our goal in this case study aims at improving the testing cost-effectiveness of an industrial scenario testing process under time constraints. We proposed a value-based testing prioritization strategy which allows tests to be ranked by how well the tests can reduce risk exposure. Combining this with the tests’ relative costs enables them to be prioritized in terms of return on investment (ROI) or risk reduction leverage (RRL). Besides, a new metric Average Percentage of Business Importance Earned (APBIE) is proposed to measure how quickly the testing can reduce the quality uncertainty and earn the relative business importance of the system under test (SUT).The results from one case study to prioritize operational testing scenarios in Galorath Inc. show that our method can improve the testing cost-effectiveness in terms of APBIE.