Assessing the detection of lateral movement through unsupervised learning techniques

Lateral movement (LM) is an umbrella term for techniques through which attackers spread from an entry point to the rest of the network. Typically, LM involves both pivoting through multiple systems and privilege escalation. As LM techniques proliferate and evolve, there is a need for advanced security controls able to detect and possibly nip such attacks in the bud. Based on the published literature, we argue that although LM-focused intrusion detection systems have received considerable attention, a prominent issue remains largely unaddressed. This concerns the detection of LM through unsupervised machine learning (ML) techniques. This work contributes to this field by capitalizing on the LMD-2023 dataset containing traces of 15 diverse LM attack techniques as they were logged by the system monitor (Sysmon) service of the MS Windows platform. We provide a panorama of this sub-field and associated methodologies, exploring the potential of standard ML-based detection. In further detail, in addition to analyzing feature selection and preprocessing, we detail and evaluate a plethora of unsupervised ML techniques, both shallow and deep. The derived scores for the best performer in terms of the AUC and F1 metrics are quite promising, around 94.7%/93% and 95.2%/93.8%, for the best shallow and deep neural network model, respectively. On top of that, in an effort to further improve on those metrics, we devise and evaluate a two-stage ML model, surpassing the previous best score by approximately 3.5%. Overall, to our knowledge, this work provides the first full-blown study on LM detection via unsupervised learning techniques, therefore it is anticipated to serve as a groundwork for anyone working in this timely field.