Log Anomaly Detection Method based on Hybrid Transformer-BiLSTM Models

Abstract

Log analysis is quite significant for reliability issues in large cloud data centers. There are noticeable problems in log anomaly detection, such as single feature extraction, unsatisfactory anomaly detection effect. In this paper, we propose a novel log anomaly detection method, which could be divided into two related parts. First, a dataset partitioning method is proposed, named K-fold Sub Hold-out Method (KSHM), which is built on the features of logs to preserve the temporality of training data when sampling. KSHM could enhance the effectiveness of sampling without increasing the number of samples, and change the way the model is trained. Second, an anomaly detection model based on hybrid Transformer-BiLSTM (TFBL) is well constructed, which could extract both temporal and semantic features of logs to serve as a source of features for comprehensive anomaly detection. Experiment results show that TFBL outperforms baseline methods in assessment criteria of accuracy, precision and F1-score, and our log anomaly detection method based on integrated KSHM and TFBL also has better anomaly detection performence.