PU Learning in Payload-based Web Anomaly Detection

Abstract

Intrusion detection is one of the most important methods for protecting web-based applications. Most anomaly detection approaches have weak detection capabilities for a new type of malicious web traffic. Besides, the misuse detection methods are based on malicious pattern matching, where the patterns usually depended on security experts. Although some supervised techniques have been applied, in real scenarios, HTTP traffic dataset is impure and more diverse. In this paper, we propose a new web anomaly detection method that combines with supervised learning model and PU learning (Positive and Unlabeled learning) based on HTTP payload data. In order to represent as many data patterns as possible, we vectorize HTTP request payloads by its numeric ASCII or Unicode value on byte-level, and each HTTP payload will be represented as a dimension-fixed numerical vector. First, our approach trains a base supervised XG-Boost model to learn the most of known attack traffics, and then the remaining normal traffics will be passed to a classifier based on the PU learning algorithm for finding some unknown malicious traffics. We test our model on a dataset gathered from a well-known security enterprise and the results show that our model achieves a remarkable accuracies on known attacks detection and has a great improvement in detecting unknown malicious web traffics.