An expert-based cost estimation model for system test execution

Abstract

To execute system tests, two fundamentally different execution techniques exist: manual and automated execution. For each system test suite, one must decide how to employ those techniques (this strategy is called execution mode). Despite general conditions such as fixed testing strategies or development philosophies, almost all projects permit a wide range of possible execution modes to choose from. In industry, execution techniques are often chosen by experts based on rules of thumb, experience and best practices. Although the results are mostly tolerable, they may be not cost-effective. In retrospect, it is often unclear on what basis those decisions were made, making it difficult to assess whether they are still valid. Finally, it is hard to predict the costs for test execution beforehand. We introduce a cost model to estimate the economic impact of execution modes. Our cost model is based on expert estimations and gives additional input for testing experts in balancing pros and cons of execution modes at hand. Furthermore, it helps documenting and persists decisions during the life time of a test suite. Additionally, we report on a first case study, applying our cost model in industry.