{"title":"具有一对一标记的日志解析器","authors":"Zhang Chunyong, Xiaojing Meng","doi":"10.1109/ICICT50521.2020.00045","DOIUrl":null,"url":null,"abstract":"System logs are often used as the primary resource in data-driven methods to ensure system health and stability. The typical process of system log analysis is to first parse unstructured logs into structured data, and then apply data mining and machine learning techniques to analyze the data and build a workflow model. Existing log parsing methods focus on similar matching of log messages and log templates. We believe that the accuracy of log message parsing is the primary task of log parsing, so we propose One-to-One, a log parser that is marked one-to-one according to the rules duringthe matching process according to the token type and part of speech. Way to parse log messages online. We evaluated Oneto-One on different log sets and compared them with the three most advanced log parsing methods. The results show that our method is similar to the results of the other three methods in parsing simple logs. However, when parsing complex OpenStack logs, the accuracy can reach 98%, which is 20% higher than the best. It can parse tens of thousands of log messages per second. This method shows high efficiency and precision for all three types of test logs, and is applicable to modern system logs.","PeriodicalId":445000,"journal":{"name":"2020 3rd International Conference on Information and Computer Technologies (ICICT)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2020-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":"{\"title\":\"Log Parser with One-to-One Markup\",\"authors\":\"Zhang Chunyong, Xiaojing Meng\",\"doi\":\"10.1109/ICICT50521.2020.00045\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"System logs are often used as the primary resource in data-driven methods to ensure system health and stability. The typical process of system log analysis is to first parse unstructured logs into structured data, and then apply data mining and machine learning techniques to analyze the data and build a workflow model. Existing log parsing methods focus on similar matching of log messages and log templates. We believe that the accuracy of log message parsing is the primary task of log parsing, so we propose One-to-One, a log parser that is marked one-to-one according to the rules duringthe matching process according to the token type and part of speech. Way to parse log messages online. We evaluated Oneto-One on different log sets and compared them with the three most advanced log parsing methods. The results show that our method is similar to the results of the other three methods in parsing simple logs. However, when parsing complex OpenStack logs, the accuracy can reach 98%, which is 20% higher than the best. It can parse tens of thousands of log messages per second. This method shows high efficiency and precision for all three types of test logs, and is applicable to modern system logs.\",\"PeriodicalId\":445000,\"journal\":{\"name\":\"2020 3rd International Conference on Information and Computer Technologies (ICICT)\",\"volume\":\"2 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2020-03-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"3\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2020 3rd International Conference on Information and Computer Technologies (ICICT)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/ICICT50521.2020.00045\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2020 3rd International Conference on Information and Computer Technologies (ICICT)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICICT50521.2020.00045","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
System logs are often used as the primary resource in data-driven methods to ensure system health and stability. The typical process of system log analysis is to first parse unstructured logs into structured data, and then apply data mining and machine learning techniques to analyze the data and build a workflow model. Existing log parsing methods focus on similar matching of log messages and log templates. We believe that the accuracy of log message parsing is the primary task of log parsing, so we propose One-to-One, a log parser that is marked one-to-one according to the rules duringthe matching process according to the token type and part of speech. Way to parse log messages online. We evaluated Oneto-One on different log sets and compared them with the three most advanced log parsing methods. The results show that our method is similar to the results of the other three methods in parsing simple logs. However, when parsing complex OpenStack logs, the accuracy can reach 98%, which is 20% higher than the best. It can parse tens of thousands of log messages per second. This method shows high efficiency and precision for all three types of test logs, and is applicable to modern system logs.