A Parallel Approach of Weighted Edit Distance Calculation for Log Parsing

Abstract

For modern software systems, larger numbers of log massages have been generated every day. By analyzing these log messages with vital information such as exception reports, developers can manage and monitor software systems efficiently. Each log message in the log file consists of a fixed part (template) and a variable part, and the fixed parts of log messages with one event type are the same, while the variable part are different. LKE (Log Key Extraction), a widely used log parser for analyzing log messages, can find the fixed parts efficiently, due to the cluster strategy base on the calculation of weighted edit distance between log messages. However, it is time-consuming to calculate the weighted edit distance for large scale log files. In this paper, we proposed a parallel approach using a unique hierarchical index structure to calculate the weighted edit distance on GPU (Graph Processing Unit). GPU has an advantage of high parallelism and is suitable for intensive computing, therefore, the time required to process large-scale logs could be reduced by this approach. Experiments show that LKE parser using GPU to calculate the weighted edit distance has high efficiency and accuracy in the HDFS data set and the marine information data set.