Reliability of conformance tests

Abstract

A conformance test is a software assurance test that is applied in order to determine if specification requirements of the software are being met. It is a time-independent model, where the software object is subjected to an a priori known test suite. The reliability of the software is the probability that it will function properly for values in the input space. Because the input space is usually very large, it is impossible to sample all input values, so in order to provide better sampling coverage, the input space is partitioned into homogeneous subspaces. Samples are drawn from each subspace for testing the software. The conformance tests based on these samples are required to pass all tests in the test suite. Based on these data, the classical statistical estimate of reliability is one. Such an estimate may be unrealistic if the sample sizes are not large. Even in such a scenario a nontrivial confidence interval is provided for the reliability.