Evaluation of spectrum based fault localization tools

Abstract

Software Fault localization (SFL) is the first step in any program debugging process. For more than three decades, researchers have aggressively studied, evaluated, and proposed numerous automatic SFL techniques spanning across various families of methods such as spectrum-based, slice-based, mutation-based, etc. Another facet contributed by researchers is the practical implementation of the above techniques in the form of open-source tools, IDE plugins, extensions, etc. Examples include (but are not limited to) GZoltzar, Jaguar, and iFL4Eclipse. Previous research has established the metrics and threshold values for the adoption of SFL techniques in real-life software development. Several attempts have been made to evaluate automatic fault repair tools, and Information Retrieval (IR) based fault localization tools. Whilst Spectrum Based Fault Localization (SBFL) remains the most contributed family of SFL methods, no studies have been found which evaluate the existing SBFL tools. This paper presents a comparative theoretical assessment of selected SBFL tools by understanding the developmental dynamics involved in implementing them and establishing results that would guide the same in the future. Our research steps can briefly be summarized as systematic collection and filtering of SBFL tools and their research papers, developing a historical timeline for the same, and comparative theoretical analysis through the lens of software engineering. We theoretically determined that there is a lack of rigorous testing for the scalability and correctness of SBFL tools. While some tools are extensible with respect to the underlying algorithm used for computation, none provide flexibility in choosing the coverage collection framework. While Open-source tools have been more successful, there is a general lack in maintenance and development post initial publication. A natural progression of this work is a large-scale empirical assessment of the SBFL tools.