EncryptoVision: A dual-modal fusion-based multi-classification model for encrypted traffic recognition

With the development of security, confidentiality, and data privacy technologies, the classification of fine-grained encrypted traffic has become increasingly important. Nowadays, existing deep learning methods, including CNN, LSTM, and transformer, have shown impressive classification performance. However, many of these methods merely utilize the raw packet bytes to generate traffic representations, resulting in the potential loss of crucial information, such as dynamic traffic patterns and changes in protocols. In this paper, we propose a dual-modal fusion-based multi-classification model for encrypted traffic recognition, called EncryptoVision. Firstly, we transform the encrypted traffic data into three-channel images and incorporate a triplet attention mechanism to enhance the interaction among the three channels. Then, we use the multi-head self-attention mechanism to expand the model’s global receptive field, allowing it to capture more detailed spatial feature information. Additionally, we also leverage the learning abilities of the transformer encoder to extract temporal feature information from the traffic for long-term time series prediction. Next, we use the spatial–temporal fusion features to obtain the fine-grained features for multi-classification. Experimental results show that our model outperforms state-of-the-art models in classification performance across four real-world encrypted traffic datasets.