Anomaly detection and visualization using Fisher Discriminant clustering of network entropy

Abstract

Entropy has been widely used to quantify information for display and examination in determining network status and in detecting anomalies. Although entropy-based methods are effective, they rely on long-term network statistics. Here, we propose an approach that deduces short term observations of network features and their respective time averaged entropies. Acute changes are detected in network feature space and depicted in a visually compact information graph. First, average entropy for each feature is calculated for every second of observation. Then, the resultant short-term information measurement is subjected to first- and second-order time averaging statistics. These time-varying statistics are used as the basis of a novel approach to anomaly estimation based on the well-known Fisher linear discriminant (FLD). This process then initiates stochastic clustering to identify the exact time of the security incident or attack on the network. The proposed method is tested on real-time network traffic data collected from Ohio Universitypsilas main Internet connection. Experimentation has shown that the presented FLD based method is accurate in identifying anomalies in network feature space. Furthermore, itpsilas performance is highly robust in the presence of bursty network traffic and it is able to detect network anomalies such as BotNet, worm outbreaks, and denial of service attacks.