Making System User Interactive Tests Repeatable: When and What Should we Control?

System user interactive tests are widely used to evaluate the behavior of an application as a whole. To automate this process, many techniques are proposed whose effectiveness are evaluated by metrics such as code coverage and fault detection. However, most of previous work assumes determinism in the outputs of interactive tests. In this paper, we propose three layers of testing outputs to examine: the code layer (codecoverage), the behavioral layer (invariant detection) and the user interaction layer (fault detection with GUI oracle). We further study the impact of common set of factors such as operating system, Java version, initial starting state and time delay on these metrics. A comprehensive experiment has been conducted on Java Swing applications, and the results show that as many as184 lines can be covered differently and up to 96% false positives with respect to fault detection. We plan to study the repeatability of interactive tests on the Android platform.