Software reliability growth models: assumptions vs. reality

Software reliability growth models are often differentiated by assumptions regarding testing and defect repair. In this paper, these model assumptions are compared to Tandem's software development and test environment. The key differences between our environment and the standard model assumptions are that (1) the total number of defects can increase due to new code being introduced during system test, but the models normally assume a constant total number of defects, and (2) the defect-finding efficiency of tests can vary but is assumed constant by the models. In spite of the model assumption violations, we (and other practitioners) continue to use the models because they are easy to apply and because the results seem reasonable. However, we are concerned about the potential inaccuracy of the models and would like to determine the effect of the assumption violations. This paper contains suggestions for research to quantify the model inaccuracy and help practitioners make accuracy vs. model complexity tradeoffs.