Power-ASTNN: A deobfuscation and AST neural network enabled effective detection method for malicious PowerShell Scripts

Abstract

PowerShell is frequently utilized by attackers in the realm of Windows system security, particularly in cyberattack activities such as information stealing, vulnerability exploitation, and password cracking. To evade detection, attackers often employ code obfuscation techniques on their scripts. Current detection solutions face challenges due to limited deobfuscation methods and a predominant focus on identifying static and local features. This limitation hinders the ability to capture fine-grained code features and long-distance semantic relationships, resulting in reduced robustness and accuracy. To address these issues, this paper presents a novel malicious script detection method, Power-ASTNN, which integrates deobfuscation and a tree neural network. Initially, the method utilizes AMSI memory dump to deobfuscate PowerShell scripts, yielding fully deobfuscated samples. Subsequently, a subtree splitting algorithm tailored for abstract syntax trees extracts fine-grained code features from subtree fragments. Finally, a two-layer neural network model encodes representations based on subtree node semantics and sequence semantics, effectively capturing the semantic characteristics of the code. Experimental results demonstrate the effectiveness of Power-ASTNN, achieving an accuracy of 98.87% on a self built dataset collected from multiple publicly available sources, while maintaining a low false negative rate and a high area under the curve (AUC) value exceeding 0.995. Furthermore, Power-ASTNN demonstrates superior detection performance against adversarial samples compared with existing detection models.