Specification-guided Golden Run for Analysis of Robustness Testing Results

Abstract

Comparison with a golden run is commonly used as an oracle in robustness testing based on fault injection. However, traditional comparison algorithms present, among other limitations, requires the system under test to present, for the same workload, the same behavior, either in presence or in absence of faults. We present an approach that uses a pair wise sequence alignment algorithm in the comparison allowing faulty traces to have some regions of dissimilarity regarding the golden run. This is possible because the algorithm is based on inexact matching and aggregates to the search semantic aspects based on SUT specification. The approach can obtain the degree of similarity with the golden run, and visually presents similarities and differences, which helps in diagnosing. This paper illustrates the application of the approach on traces collected during robustness testing of the Wireless Transaction Protocol (WTP).