An Empirical Comparison of Two Different Strategies to Automated Fault Detection: Machine Learning Versus Dynamic Analysis

Abstract

Software testing is an established method to ensure software quality and reliability, but it is an expensive process. In recent years, the automation of test case generation has received significant attention as a way to reduce costs. However, the oracle problem (a mechanism for determine the (in) correctness of an executed test case) is still major problem which has been largely ignored. Recent work has shown that building a test oracle using the principles of anomaly detection techniques (mainly semisupervised/ unsupervised learning models based on dynamic execution data consisting of an amalgamation of input/output pairs and execution traces) is able to demonstrate a reasonable level of success in automatically detect passing and failing execution [1], [2]. In this paper, we present a comparison study between our machine-learning based approaches and an existing techniques from the specification mining domain (the data invariant detector Daikon [3]). The two approaches are evaluated on a range of midsized systems and compared in terms of their fault detection ability. The results show that in most cases semi-supervised learning techniques perform far better as an automated test classifier than Daikon. However, there is one system for which our strategy struggles and Daikon performed far better. Furthermore, unsupervised learning techniques performed on a par when compared with Daikon in several cases.