Detection of anomalies in the HDFS dataset

Abstract

Big data systems are stable enough to store and process large volumes of quickly changing data. However, these systems are composed of massive hardware resources, which can easily cause their subcomponents to fail. Fault tolerance is a key attribute of such systems as they maintain availability, reliability and constant performance during failures. Implementing efficient fault-tolerant solutions in big data presents a challenge because fault tolerance has to satisfy some constraints related to system performance and resource consumption. To protect online computer systems from malicious attacks or malfunctions, log anomaly detection is crucial. This paper provides a new approach to identify anomalous log sequences in the HDFS (Hadoop Distributed File System) log dataset using three algorithms: Logbert, DeepLog and LOF. Then, it assess performance of all algorithms in terms of accuracy, recall, and F1-score.