A New Approach to Building a Multi-tier Direct Access Knowledgebase for IDS/SIEM Systems

Looking at current IDS and SIEM systems, we observe heavy processing power dedicated solely to answering a simple question, What is the format of the log line that the IDS (or SIEM) system should process next? Due to the apparent difficulties of uniquely identifying a log line at run-time, most systems today do little or no normalisation of the events they receive. Indeed these systems often rely on popular search engine applications for processing and analysing the event information they receive, which results in slower and far less accurate event correlations. In this process, a large list of tokenisers is usually created in order to find an answer to the above posted question. The tokenisers are run against the log lines, until a match is found. The appropriate log line can then be passed on to the correct extraction module for further processing. This process is currently the standard procedure of most IDS and SIEM systems. To address this problem and to optimise and improve the said process, this paper describes a method for detecting the exact type and format of a read log line in the first place. The method presented performs in an efficient manner, while it is less resource hungry. The proposed detection system is described and implemented, its pros and cons are analysed and weighed against methods currently implemented by popular IDS and SIEM systems for solving this task.