Interpreting Coverage Information Using Direct and Indirect Coverage

Abstract

Because of the numerous benefits of tests, developers often wish their applications had more tests. Unfortunately, it is challenging to determine what new tests to add in order to improve the quality of the test suite. A number of approaches, including numerous coverage criteria, have been proposed by the research community to help developers focus their limited testing resources. However, coverage criteria often fall short of this goal because achieving 100% coverage is often infeasible, necessitating the difficult process of determining if a piece of uncovered code is actually executable, and the criteria do not take into account how the code is covered. In this paper, we propose a new approach for interpreting coverage information, based on the concepts of direct coverage and indirect coverage, that address these limitations. We also presents the results of an empirical study of 17 applications that demonstrate that indirectly covered code is common in real world software, faults in indirectly covered code are significantly less likely to be detected than faults located in directly covered code, and indirectly covered code typically clusters at the method level. This means that identifying indirectly covered methods can be effective at helping testers improve the quality of their test suites by directing them to insufficiently tested code.