PHP-based malicious webshell detection based on abstract syntax tree simplification and explicit duration recurrent networks

Malicious webshells are the most common attack scripts used by attackers in web penetration. Attackers typically obfuscate strings of PHP-based malicious webshells and encrypt communication traffic to bypass security devices. In this case, the opcode sequences of the PHP-based malicious webshells become excessively long and contain many irrelevant features, which affect the efficacy of the detection method. This study proposes a new PHP-based malicious webshell detection method. The proposed method introduces three simplification strategies for the three main types of nodes in the abstract syntax trees of PHP scripts to reduce the length and noise of opcode sequences of PHP-based malicious webshells. An explicit duration recurrent network (EDRN), a recurrent neural network based on an extended hidden semi-Markov model, is used to detect malicious webshells. Word2vec is adopted to convert the opcode sequences of the PHP scripts into vectors that serve as the input for the EDRN. Experiments were conducted using public datasets collected from GitHub. The experimental results indicated that EDRN outperformed popular recurrent neural networks. The proposed method demonstrated superior performance compared with several state-of-the-art approaches and mainstream tools, achieving an accuracy of 0.993, an F1 score of 0.990, and a recall rate of 0.991. When only 20% of the datasets were used for training, the proposed method achieved accuracy, recall, and F1 scores of 0.985, 0.983, and 0.980, respectively, significantly outperforming existing approaches.