ST-MemA: Leveraging Swin Transformer and memory-enhanced LSTM for encrypted traffic classification

Traffic classification is essential for effective intrusion detection and network management. However, with the pervasive use of encryption technologies, traditional machine learning-based and deep learning-based methods often fall short in capturing the fine-grained details in encrypted traffic. To address these limitations, we propose a memory-enhanced LSTM model based on Swin Transformer for multi-class encrypted traffic classification. Our approach first reconstructs raw encrypted traffic by converting each flow into single-channel images. A hierarchical attention network, incorporating both byte-level and packet-level attention, then performs comprehensive feature extraction on these traffic images. The resulting feature maps are subsequently classified to identify traffic flow categories. By combining the long-term dependency capabilities of LSTM with the Swin Transformer’s strengths in feature extraction, our model effectively captures global features across diverse traffic types. Furthermore, we enhance LSTM with memory attention, enabling the model to focus on more fine-grained information. Experimental results on three public datasets—USTC-TFC2016, ISCX-VPN2016, and CIC-IoT2022 show that our model, ST-MemA, improves the classification accuracy to 99.43%, 98.96% and 98.21% and $F_1$-score to 0.9936, 0.9826 and 0.9746, respectively. The results also demonstrate that our proposed model outperforms current state-of-the-art models in classification accuracy and computational efficiency.