A log-based anomaly detection method with the NW ensemble rules

Abstract

Log analysis can be used for software system anomaly detection, and ensemble learning can handle log data with imbalanced characteristics. Therefore, log-based anomaly detection with ensemble learning is a good choice. However, the existing data balancing methods used in ensemble learning may destroy the distribution of the original log data and affect the accuracy of the anomaly detection results. Besides, the existing ensemble rules do not take into account the relationship between the samples to be detected and the historical log data. Therefore, we propose a log-based anomaly detection method with the NW (Neighbor Weighting) ensemble rules, which uses a data balancing method based on spectral clustering so that the balanced log data can maintain the distribution of the original data and meet the quantity balance at the same time. Then, a new group of ensemble rules is proposed and used for anomaly detection with higher accuracy. We performed experiments on six large log data sets with different types of systems and verified the feasibility and universality of the method in this paper.