An Efficient Massive Log Discriminative Algorithm for Anomaly Detection in Cloud

Log anomaly detection is a critical step towards building a secure and trustworthy cloud system. As more corporations turn to cloud system to store and process their most valuable data, the risk of a potential breach of those systems increases exponentially. However, conventional top-n log candidates anomaly detection methods, such as Deeplog and N-gram, often suffer from the limited scope of the top-n list, which rules out many potentially suitable candidates. In this paper, we propose Discounted Cumulative Gain (DCG) discriminative algorithm that ranks all the log candidates and calculates the dcg score to determine the number of log candidates. To demonstrate the effectiveness of our algorithm, we conduct comprehensive experiments under different log workloads. Experimental evaluations show that DCG has outperformed Deeplog and N-gram methods in cloud systems, and improved the F-score of Deeplog and N-gram by up to 3.8% and 11.6% respectively.