DMSE: An efficient malicious traffic detection model based on deep multi-stacking ensemble learning

In the context of increasing cyber threats, developing an efficient malicious traffic detection model to recognize the cyber attacks has become an urgent demand in the field of cyber security. This paper proposes an efficient malicious traffic detection model called DMSE based on deep multi-stacking ensemble learning, it is primarily consisted of feature representation module, base model detection module and multi-stacking ensemble learning module. In the feature representation phase, we propose a novel RGB image representation method, which hierarchically represents the global and local features of network traffic by allocating the information to three channels of RGB images. In the base model detection phase, we adopt five different deep learning models—CNN, TCN, LSTM, BiLSTM and BiTCN—as base models for the first-stage prediction. In the multi-stacking ensemble learning phase, we adopt the best-performing BiTCN from extensive experiments as the meta-learner to perform a second prediction using the results from the first stage, thereby obtaining the final detection result. Experiments conducted on USTC-TFC2016, CTU and ISAC datasets demonstrate that the proposed DMSE model significantly outperforms existing ensemble learning-based detection models in terms of accuracy, F1-score, false positive rate (FPR), true positive rate (TPR) and stability. The experimental results indicate that the proposed DMSE model can effectively identify and defend against network attacks, providing the new perspectives and technical support for maintaining a secure network environment.