Malware Family Characterization with Recurrent Neural Network and GHSOM Using System Calls

Abstract

Nowadays, a massive amount of sensitive data which are accessible and connected through personal computers and cloud services attract hackers to develop malicious software (malware) to steal them. Owing to the success of deep learning on image and language recognition, researchers direct security systems to analyze and identify malware with deep learning approaches. This paper addresses the problem of analyzing and identifying complex and unstructured malware behaviors by proposing a framework of combining unsupervised and supervised learning algorithms with a novel sequence-aware encoding method. Particularly, a hybrid GHSOM (the Growing Hierarchical Self-Organizing Map) algorithm is proposed to cluster and encode similar malware behavior sequences from system call sequences to clustering feature vectors. Then, a Recurrent Neural Network (RNN) is trained to detect malware and predict their corresponding malware families based on the sequence of the behavior vectors. Our experiments show that the accuracy rate can be up to 0.98 in malware detection and 0.719 in malware classification of an 18-category malware dataset.