Coverage Metrics and Detection of Injection Vulnerabilities: An Experimental Study

Coverage is frequently considered a metric of the quality of the tests and, consequently, of the software dependability. Although one tends to assume a similar relation in the context of vulnerability detection, such assumption is yet to be shown in practice. Although the effectiveness of vulnerability detection tools is limited and largely dependent on the context, developers usually select and use a single tool and implicitly trust on its results. In this practical experience report we study the relation between coverage measurements and the quality of the results of detection tests for injection vulnerabilities, in particular SQL Injection, considering two state of the art tools and multiple testing configurations. Such relation is of utmost importance for developers to understand how good vulnerability detectors are and to compare alternative tools. Results show that code coverage is indeed an effective mean to estimate the quality of vulnerability detection tests and is useful to compare different sets of tests. However, they also show that domain specific metrics are much more effective than generic ones.