LogSayer: Log Pattern-driven Cloud Component Anomaly Diagnosis with Machine Learning

Abstract

Anomaly diagnosis is a critical task for building a reliable cloud system and speeding up the system recovery form failures. With the increase of scales and applications of clouds, they are more vulnerable to various anomalies, and it is more challenging for anomaly troubleshooting. System logs that record significant events at critical time points become excellent sources of information to perform anomaly diagnosis. Never-theless, existing log-based anomaly diagnosis approaches fail to achieve high precision in highly concurrent environments due to interleaved unstructured logs. Besides, transient anomalies that have no obvious features are hard to detect by these approaches. To address this gap, this paper proposes LogSayer, a log pattern-driven anomaly detection model. LogSayer represents the system state by identifying suitable statistical features (e.g. frequency, surge), which are not sensitive to the exact log sequence. It then measures changes in the log pattern when a transient anomaly occurs. LogSayer uses Long Short-Term Memory (LSTM) neural networks to learn the historical correlation of log patterns and applies a BP neural network for adaptive anomaly decisions. Our experimental evaluations over the HDFS and OpenStack data sets show that LogSayer outperforms the state-of-the-art log-based approaches with precision over 98%.