USTEP: Unfixed Search Tree for Efficient Log Parsing

Abstract

Logs record valuable system information at runtime. They are widely used by data-driven approaches for development and monitoring purposes. Parsing log messages to structure their format is a classic preliminary step for log-mining tasks. As they appear upstream, parsing operations can become a processing time bottleneck for downstream applications. The quality of parsing also has a direct influence on their efficiency. Previous approaches toward online log parsing focused on stateful methods. But an increasing number of tasks ask for real time monitoring. Regarding this problem, we propose USTEP, an online log parsing method based on an evolving tree structure. Evaluation results on a panel of 13 datasets coming from different real-world systems demonstrate USTEP superiority in terms of both effectiveness and robustness when compared to other online methods. We also introduce USTEP-UP, a way of running multiple decentralized instances of USTEP in parallel.