Evaluating the Effectiveness of BEN in Localizing Different Types of Software Fault

Abstract

Debugging or fault localization is one of the most challenging tasks during software development. Many tools have been developed to reduce the amount of effort and time software developers have to spend on fault localization. In this paper, we evaluate the effectiveness of a fault localization tool called BEN in localizing different types of software fault. Assuming that combinatorial testing has been performed on the subject program, BEN leverages the result obtained from combinatorial testing to perform fault localization. Our evaluation focuses on impact of three properties of software fault on the effectiveness of BEN. The three properties include accessibility, input value sensitivity and control flow sensitivity. A random test set-based approach is used to measure the three properties. The experimental results suggest that BEN is more effective, respectively, in localizing faults of lower accessibility, input value-insensitive faults or control flow-insensitive faults than localizing faults of higher accessibility, input value-sensitive or control flow-sensitive faults in the subject programs. The insights obtained from our evaluation can be applied to other fault localization tools that are similar to BEN, and can be used to identify opportunities for further research on combinatorial testing-based fault localization.