Measuring the effectiveness of modern security products to detect and contain emerging threats — A consensus-based approach

Abstract

Increasingly the idea that cyber-attacks can be stopped at the periphery of the network has become a fool's errand. In today's computing environment and cyber-threat landscape, individuals as well as corporations have recognized the fact that (i) with the emergence of cloud based computing there are no longer network boundaries under your control that can be protected, (ii) threats are often distributed in nature both in time and space - making detection extremely difficult, and (iii) the working assumption is not that you can prevent infections (the goal of 100% prevention is no longer practical) but rather, given that your "system" will be compromised, how quickly can you detect the breach and how do you minimize the impact of such an event. In this new environment, the idea that measuring the number of infected files detected within end-point devices is a good measure of the effectiveness of Anti-Malware and Security related products seems foolish. Instead, the industry has recognized that time to detect, time to countermeasure issuance, and ability to identify short-lived C&C sites are more relevant to determining the "goodness" of security products. Within this context, the authors have undertaken to develop benchmark metrics to test the ability of commercial automated gateway and endpoint security services to classify and categorize different types of web traffic (malicious content, malicious activity, non-malicious category). A test methodology has been developed for this purpose, based on the Wireless Systems Security Research Laboratory (WSSRL) test methodology, and extensions to CheckVir Battery Test. Using this methodology, eight gateway protection services were tested and classified for their ability to identify the incoming traffic as malicious, C&C communications, and non-malicious content. A key component of the methodology is the concept of eventual consensus, a methodology whereas new threats are classified as malicious or not when (n/2+ 1) security products agree on the nature of the threat over time. The methodology was developed as a simplified extension of the well known Byzantine Agreement protocol first discussed by Leslie Lamport.