Towards a Bayesian Network Model for Predicting Flaky Automated Tests

Abstract

Artificial intelligence and machine learning are making it possible for computers to diagnose some medical diseases more accurately than doctors. Such systems analyze millions of patient records and make generalizations to diagnose new patients. A key challenge is determining whether a patient's symptoms are attributed to a known disease or other factors. Software testers face a similar problem when troubleshooting automation failures. They investigate questions like: Is a given failure due to a defect, environmental issue, or flaky test? Flaky tests exhibit both passing and failing results although neither the code nor test has changed. Maintaining flaky tests is costly, especially in large-scale software projects. In this paper, we present an approach that leverages Bayesian networks for classifying and predicting flaky tests. Our approach views the test flakiness problem as a disease by specifying its symptoms and possible causes. Preliminary results from a case study suggest the approach is feasible.