PosParser: A Heuristic Online Log Parsing Method Based on Part-of-Speech Tagging

Abstract

Log parsing, the process of transforming raw logs into structured data, is a key step in the complex computer system's intelligent operation and maintenance and therefore has received extensive attention. Among all log parsing methods, heuristic log parsing methods are lightweight and can work in a streaming mode to well meet the real-time parsing requirements. However, the existing log representations used in the heuristic log parsing methods are not powerful in distinguishing log messages, which leads to low parsing accuracy and weak generality. Inspired by trigger word extraction of the event detection task in natural language processing (NLP), this paper proposes an online log parser, named PosParser, which employs the part-of-speech (PoS) tagging to extract a function token sequence (FTS) as the log message representation, and then identify event templates of log messages through the FTS. Experimental results on sixteen logs from real systems demonstrate that the FTS is powerful in distinguishing log messages from different event templates, and PosParser not only performs better in terms of parsing accuracy than state-of-the-art methods but is also comparable to them in efficiency.