Identifying malicious traffic under concept drift based on intraclass consistency enhanced variational autoencoder

Accurate identification of malicious traffic is crucial for implementing effective defense counter-measures and has led to extensive research efforts. However, the continuously evolving techniques employed by adversaries have introduced the issues of concept drift, which significantly affects the performance of existing methods. To tackle this challenge, some researchers have focused on improving the separability of malicious traffic representation and designing drift detectors to reduce the number of false positives. Nevertheless, these methods often overlook the importance of enhancing the generalization and intraclass consistency in the representation. Additionally, the detectors are not sufficiently sensitive to the variations among different malicious traffic classes, which results in poor performance and limited robustness. In this paper, we propose intraclass consistency enhanced variational autoencoder with Class-Perception detector (ICE-CP) to identify malicious traffic under concept drift. It comprises two key modules during training: intraclass consistency enhanced (ICE) representation learning and Class-Perception (CP) detector construction. In the first module, we employ a variational autoencoder (VAE) in conjunction with Kullback-Leibler (KL)-divergence and cross-entropy loss to model the distribution of each input malicious traffic flow. This approach simultaneously enhances the generalization, interclass consistency, and intraclass differences in the learned representation. Consequently, we obtain a compact representation and a trained classifier for non-drifting malicious traffic. In the second module, we design the CP detector, which generates a centroid and threshold for each malicious traffic class separately based on the learned representation, depicting the boundaries between drifting and non-drifting malicious traffic. During testing, we utilize the trained classifier to predict malicious traffic classes for the testing samples. Then, we use the CP detector to detect the potential drifting samples using the centroid and threshold defined for each class. We evaluate ICE-CP and some advanced methods on various real-world malicious traffic datasets. The results show that our method outperforms others in identifying malicious traffic and detecting potential drifting samples, demonstrating outstanding robustness among different concept drift settings.