A Toolkit for Testing Stochastic Simulations against Statistical Oracles

Abstract

Stochastic simulations are developed and employed across many fields, to advise governmental policy decisions and direct future research. Faulty simulation software can have serious consequences, but its correctness is difficult to determine due to complexity and random behaviour. Stochastic simulations may output a different result each time they are run, whereas most testing techniques are designed for programs which (for a given set of inputs) always produce the same behaviour. In this paper, we introduce a new approach towards testing stochastic simulations using statistical oracles and transition probabilities. Our approach was implemented as a toolkit, which allows the frequency of state transitions to be tested, along with their final output distribution. We evaluated our toolkit on eight simulation programs from a variety fields and found it can detect errors at least three times smaller (and in one case, over 1000 times smaller) than a conventional (tolerance threshold) approach.