Using approximate matching and machine learning to uncover malicious activity in logs

Abstract

The rapid expansion of digital services has led to an unprecedented surge in digital data production. Logs play a critical role in this vast volume of data as digital records capture notable events within systems or processes. Large-scale systems generate an overwhelming number of logs, making manual examination by analysts infeasible during critical events or attacks. While hashes, whether cryptographic or fuzzy, are widely used in digital forensics because they serve as the foundation for software integrity and validation, authentication and identification, similarity analysis, and fragment detection, this study investigates and extends the use of approximate matching (AM) algorithms in semi-structured data, such as logs. Existing AM algorithms such as ssdeep, sdhash, TLSH, and LZJD struggle particularly with semi-structured data due to the size of the input data being comparatively small, with syntactical and structural information comprising a significant amount of the data. We present a novel approximate matching algorithm for application across a range of semi-structured data types, which requires no knowledge of the underlying data structure. The algorithm produces digests that serve as input to a machine learning classifier, classifying the behaviour of the underlying logs the hashes represent. Experimental results on a benchmark dataset of IoT network traffic show that the proposed framework can correctly discern malicious logs from benign records with a 95% accuracy, with an F1 score of 0.98. The behaviour of the records deemed malicious was then correctly identified with a 99% accuracy when evaluated using a test data set, producing an average F1 score of 0.99. Additionally, we demonstrate that this approach provides a faster and lightweight framework to perform classification with high accuracy on a list of logs, producing those indicative of an attack for review.