ConceptUML: Multiphase unsupervised threat detection via latent concept learning, Hidden Markov Models and topic modelling

Abstract

Detecting lateral movement threats in large-scale system logs is a critical challenge due to the scarcity of labelled attack data, the presence of imbalanced datasets, and the sophisticated nature of modern adversaries. To address these issues, we propose ConceptUML, a semantic-driven, fully unsupervised threat detection framework designed to automatically identify anomalies related to lateral movement in heterogeneous log data. ConceptUML is structured around a three-phase architecture. In Phase 1 (Latent Semantic Learning), contextualized embeddings generated by Sentence-BERT are combined with Non-negative Matrix Factorization to extract abstract concepts from system logs and external threat intelligence sources such as MITRE ATT&CK and CAPEC. In Phase 2 (Unsupervised Threat Detection), a Hidden Markov Model is applied to cluster logs based on learned concepts, and each cluster is scored according to its semantic similarity to known adversarial techniques. Phase 3 (Decision Refinement) uses topic modelling to further isolate malicious event log subsets from within suspicious clusters, enabling high-precision triage. We evaluate ConceptUML using four real-world event log datasets, including Windows Event Logs and multiple subsets of the LMD-23 dataset, encompassing attacks such as exploitation of hashing techniques and remote services. The enhanced model with topic modelling achieves up to 92.54% detection quality and reduces detection error to as low as 8.14%, outperforming several baseline approaches including AutoEncoder, LogAnomaly, LOF, and DBScan. Our results confirm that ConceptUML delivers interpretable, scalable, and highly effective detection of lateral movement threats without requiring labelled training data or extensive manual feature engineering.