A methodology for quantitative evaluation of software reliability using static analysis

Abstract

This paper proposes a methodology for quantitative evaluation of software reliability in updated COTS or Open Source components. The model combines static analysis of existing source code modules, limited testing with execution path capture, and a series of Bayesian Belief Networks. Static analysis is used to detect faults within the source code which may lead to failure. Code coverage is used to determine which paths within the source code are executed as well as their execution rate. A series of Bayesian Belief Networks is then used to combine these parameters and estimate the reliability for each method. A second series of Bayesian Belief Networks then combines the module reliabilities to estimate the net software reliability. A proof of concept for the model is provided, as the model is applied to five different open-source applications and the results are compared with reliability estimates using the STREW (Software Testing and Early Warning) metrics. The model is shown to be highly effective and the results are within the confidence interval for the STREW reliability calculations, and typically the results differed by less than 2%. This model offers many benefits to practicing software engineers. Through the usage of this model, it is possible to quickly assess the reliability of a given release of a software module supplied by an external vendor to determine whether it is more or less reliable than a previous release. The determination can be made independent of any knowledge of the developer's software development process and without any development metrics.