Coverage Evaluation Through Fault Injection: Fault Sampling And Statistical Analysis

Imperfect coverage is known to drastically reduce the effectiveness of redundancy and yet the determination of this parameter has been elusive. Fault injection simulation has been recognized as an effective technique to collect data for the statistical estimation of the coverage parameter. In a fault simulation experiment, faults are sampled from the fault population and then injected into, the system’s simulation model to determine the number of faults that are uncovered by the system’s fault detection mechanisms, i.e., the number of faults that cause unsafe outputs. When no unsafe failure occurs during a large number of simulation runs (i.e., when a large number of faults are injected), it is not possible to quantify the exact coverage since the only available information is that the system has not yet failed in the presence of the chosen faults. However, it is possible, with a high confidence level, to infer that the coverage value reach at least some target value, i.e., to quantify a lower bound on the safety measure. Both the confidence level and the lower bound depend on the test duration (i.e., sample size) and the way in which faults are selected from the fault population. For simple random sampling, where every fault has an equal chance of being selected, we present a formula that can be used to predict the minimum number of fault injections for a target lower bound on coverage. For ultra-reliable and safety-critical systems, the desired coverage is supposedly extremely close to 1. In this case, our result shows that the minimum