Experiments in software reliability estimation

Abstract

There is an urgent need for manufacturers and users of programmable electronic systems to be able to quantify the risk of system failure due to design faults in software. The reliability of the hardware components of such systems can be assessed using well-tried techniques. By contrast, software reliability is still a ‘grey area’, with no generally accepted methods of assessment.

This paper describes the results of using the Littlewood Stochastic Reliability Growth Model with maximum likelihood parameter estimation to forecast the behaviour of sets of simulated failure data, generated on the assumptions of the model and using a variety of parameter values. The forecasts are long-term, such as would be made for large software products whose reliability is important from the support cost point of view, but not critical as regards safety. The data is of the ‘grouped’ variety: counts of faults found in successive intervals.

The predictions are generally of low accuracy. They are particularly bad for extreme parameter values, corresponding to very many, very infrequently manifest faults, and to few frequently manifest faults. The length of the period of observation relative to the average rate of fault manifestation is also crucial.

Possible reasons for this poor performance and improvements to the estimation methods are discussed.