LogGzip: Towards log Parsing with lossless compression

Abstract

Automated analysis of complex logs from Internet of Things(IoT) devices facilitates failure diagnosis and system status monitoring. Log parsing, the first step in this process, converts raw logs into structured data. Due to the vast size and intricate structure of IoT system logs, parsers must effectively handle various log formats. Supervised learning parsers require labor-intensive manual data labeling. Clustering-based parsers, as an unsupervised method, minimize expert involvement and manual annotation. However, existing clustering-based parsers struggle with the diverse formats of log data and handling minor variations or noise within logs, due to their reliance on specific log structures or the need to transform logs into particular representations. To address the above problems, the paper proposes LogGzip, a clustering log parser based on the gzip lossless compressor. It employs a gzip compressor to measure differences in compressed lengths between logs to identify the complex patterns and regularities in the logs, and designs compression distance calculation method to construct a distance matrix as a measure of log event similarity. At the same time, the overhead in the compression process is reduced by building a compression dictionary. Finally, clustering analysis is performed using the similarity scores. Experimental results demonstrate that the parsing accuracy of LogGzip outperforms the existing state-of-the-art log parsers.