A significant improvement for anti-malware tests

Abstract

Despite ongoing improvements in the quality of antimalware tests, the way in which test results are handled often shows a low level of sophistication. In this paper, we introduce the simple concept of confidence intervals and statistical significance to these tests, and show that many of the “best practice” approaches common in other fields are lacking in the security-software testing industry. Further, we argue that the lack of these techniques harms the industry as a whole, and provide a road map for broader adoption of well-known statistical techniques for estimating the confidence interval on measurements.