Unsupervised Multi-Target Cross-Service Log Anomaly Detection

Abstract

Log analysis, especially log anomaly detection, can help debug systems and analyze root causes to provide reliable services. Deep learning is a promising technology for log anomaly detection. However, deep learning methods need a large amount of training data, which is hard for a newly deployed system to collect sufficient logs. Transfer learning becomes a possible method to solve the problem that can apply the knowledge from a long-term deployed system (source) to a newly deployed system (target). Existing transfer learning methods focus on transferring the knowledge from a source system to a single target system within the same service, in which the source and the target belong to the same service (e.g. operating system, supercomputer, or distributed system). They achieve low performance when applied to multiple target and different services systems because of the obvious differences in log format, syntax, semantics, and component call between different services and the individual training of multiple models for each target system. To tackle the problems, we propose an unsupervised multi-target cross-service log anomaly detection method based on transfer learning and contrastive learning (LogMTC). LogMTC exploits contrastive learning to learn a single model on combined data from the source and multiple target systems, which can fit multiple target systems simultaneously and improve efficiency. LogMTC exploits a hypersphere loss and two contrastive losses to minimize the feature differences crossing different services. Our experiments on two services (supercomputer and distributed system) and three log datasets show that our method is superior to the existing transfer learning methods in the same service, cross-service, and multi-target log anomaly detection. Compared with the best peer accurate transfer learning algorithm LogTAD, LogMTC improves 1.14%–8.28$\%$ F1 score in multi-target transfer and is 1.12–1.22 times faster.