KAD: a knowledge formalization-based anomaly detection approach for distributed systems

Abstract

Large-scale distributed systems are becoming key engines of the IT industry due to their scalability and extensibility. A distributed system often involves numerous complex interactions among components, suffering anomalies such as data inconsistencies between components and unanticipated delays in response times. Existing anomaly detection techniques, which extract knowledge from system logs using either statistical or machine learning techniques, exhibit limitations. Statistical techniques often miss implicit anomalies that are related to complex interactions manifested by logs, whereas machine learning techniques lack explainability and they are usually sensitive to log variations. In this paper, we propose KAD, a knowledge formalization-based anomaly detection approach for distributed systems. KAD includes a general knowledge description language (KDL), leveraging the general structure of system logs and extended Backus-Naur form (EBNF) for complex knowledge extraction. Particularly, the semantic set is constructed based on the bidirectional encoder representation from the transformer (BERT) model to improve the expressive capabilities of KDL in knowledge description. In addition, KAD incorporates distributed scheduling computation module to improve the efficiency of anomaly detection processes. Experimental results based on two widely used benchmarks show that KAD can accurately describe the knowledge associated with anomalies, with a high F1-score in detecting various anomaly types.