A prototypical alignment approach to unknown traffic classification using BERT

Abstract

As encrypted internet traffic continues to increase, classifying previously unseen traffic has become a major challenge in real-world network environments. Traditional traffic classification models are designed for closed-world scenarios and struggle to process novel traffic types. To address this in real-world network environments, coarse-grained classification approaches have been proposed, which assign all unknown traffic to a single ”unknown” class. However, this coarse-grained labeling limits the model’s ability to perform the fine-grained classification of diverse and encrypted traffic behaviors. To address this, we propose Unknown Traffic Prototypical Alignment Bidirectional encoder representations from transformers (UT-PAB), a semi-supervised learning framework for the fine-grained classification of unknown traffic in open-world scenarios. UT-PAB operates in two phases: (1) a pre-training phase that learns general traffic patterns through supervised and masked token prediction tasks, and (2) a fine-tuning phase that refines representations using contrastive learning and prototype-based supervision. These two phases enable the model to cluster unknown traffic by semantic similarity without relying on protocol-specific features. We evaluated the effectiveness of UT-PAB on two benchmark datasets, ISCX-VPN and USTC-TFC, by comparing it against baseline methods based on clustering and representation learning. We conducted extensive experiments on two benchmark datasets across various unknown traffic ratios and demonstrated that the proposed method outperformed state-of-the-art methods by a minimum of 10.63%p and a maximum of 75.34%p improvement in overall accuracy.