Dynamic test allocation model for software reliability

Abstract

In any non trivial software system the reliability of the software cannot be determined exactly. Instead, we must apply statistical methods to create an estimate based on a sample of test cases. Our goal is, given a fixed total number of test cases, to determine how to allocate these test cases among the partitions of the software so as to minimize the variance incurred by the maximum likelihood estimator of the overall software reliability. In contrast to fixed sampling models, where the proportion of test cases taken from each partition is determined before reliability testing begins, we make allocation decisions dynamically throughout the testing process. We compare the results from the dynamic test allocation model with the optimal sampling model and demonstrate its strength with respect to the variance incurred when the overall system reliability is estimated by its maximum likelihood estimator both theoretically and through Monte Carlo simulations.