Variational Encoder-Decoder Recurrent Neural Network (VED-RNN) for Anomaly Prediction in a Host Environment

Abstract

Intrusion detection systems (IDS) are important security tools. NIDS monitors network's traffic and HIDS filters local one. HIDS are often based on anomaly detection. Several studies deal with anomaly detection using system-call traces. In this paper, we propose an anomaly detection and prediction approach. System-call traces, invoked by the running programs, are analyzed in real time. For prediction, we use a Sequence to sequence model based on variational encoder-decoder (VED) and variants of Recurrent Neural Networks (RNN), these architectures showed their performance on natural language processing. To make the analogy, we exploit the semantics behind the invoking order of system-calls that are then seen as sentences. A preprocessing phase is added to optimize the prediction model input data representation. A one-class classification is done to categorize the sequences into normal or abnormal. Tests are achieved on the ADFA-LD dataset and showed the advantage of the prediction for the intrusion detection/prediction task.