A methodology for designing accurate anomaly detection systems

Anomaly detection systems have the potential to detect zero-day attacks. However, these systems can suffer from high rates of false positives and can be evaded through through mimicry attacks. The key to addressing both problems is careful control of model generalization. An anomaly detection system that undergeneralizes generates too many false positives, while one that overgeneralizes misses attacks. In this paper, we present a methodology for creating anomaly detection systems that make appropriate trade-offs regarding model precision and generalization. Specifically, we propose that systems be created by taking an appropriate, undergeneralizing data modeling method and extending it using data pre-processing generalization heuristics. To show the utility of our methodology, we show how it has been applied to the problem of detecting malicious web requests.