Log-TF-IDF and NETCONF-Based Network Switch Anomaly Detection

Abstract

In this study, we propose and evaluate a model that utilizes both log data and state data to detect abnormal conditions in network switches. Building upon our previous research and drawing inspiration from TF-IDF used in natural language processing to measure word importance, we propose a statistical method, Log-TF-IDF, to quantify the rarity of each log pattern in the log data. Furthermore, based on this Log-TF-IDF, we introduce the AB Score, which quantifies how abnormal the current log pattern is. Our findings indicate that the AB Score is notably higher and more volatile in abnormal conditions. We confirm that anomaly detection is feasible through the AB Score, which has the advantage of being computationally efficient due to its statistical basis. We combined the metrics generated during the AB Score calculation with resource data collected with NETCONF and developed a machine-learning model to detect abnormal conditions in network switches. We confirm that this model can detect abnormal conditions with an F1 score of 0.86 on our collected dataset, confirming its viability for detecting abnormal states in network equipment.