ProvGOutLiner: A lightweight anomaly detection method based on process behavior features within provenance graphs

Abstract

The Provenance Graph is an effective tool for host-based intrusion detection. It uses directed graph to represent interactions between system entities and is widely used to capture and analyze system activities. Provenance graph-based anomaly detection methods aim to identify potential security threats in host environments. Compared to traditional intrusion detection techniques, provenance graph-based methods are more effective at detecting stealthy attacks. However, existing learning-based methods often rely on large amounts of labeled data. These methods have high computational costs and lack interpretability. This makes it difficult to clearly identify specific attack behaviors. To address these issues, we propose ProvGOutLiner: A lightweight and unsupervised anomaly detection method for provenance graphs. This method is based on process behavior characteristics. We analyze common attack behaviors in detail and find that the outgoing edge types and counts from processes in the provenance graph exhibit distinctive behavior patterns. Based on this observation, we introduce a Process Behavior Tree. This tree generates feature vectors for process behaviors by statistically analyzing the types and counts of outgoing edges from its nodes. We then apply a clustering algorithm to detect anomalous behaviors in an unsupervised manner. The construction of the Process Behavior Tree and feature extraction do not require complex models, which enables lightweight detection. We evaluate our method on the DARPA public dataset. The results show that ProvGOutLiner significantly reduces computational overhead while accurately identifying malicious process activities. ProvGOutLiner achieves a recall rate of 99%, a precision rate of 96%, and our method significantly reduces computation time.