Machine Learning Toolkit for System Log File Reduction and Detection of Malicious Behavior

Abstract

The increasing use of encryption blinds traditional network-based intrusion detection systems (IDS) from performing deep packet inspection. An alternative data source for detecting malicious activity is necessary. Log files found on servers and desktop systems provide an alternative data source containing information about activity occurring on the device and over the network. The log files can be sizeable, making the transport, storage, and analysis difficult. Malicious behavior may appear as normal events in logs, not triggering an error or another obvious indicator, making automated detection challenging. The research described here utilizes a Python-based toolkit approach with unsupervised machine learning to reduce log file sizes and detect malicious behavior.