ER-ERT:A Method of Ensemble Representation Learning of Encrypted RAT Traffic

Abstract

Remote Access Trojan (RAT) is one of the major threats to today's network environment. It is a class of malware frequently used by hacking collectives to monitor victims' actions and steal personal information in targeted computers. Traditional machine learning algorithms have been widely used to detect malicious encrypted RAT traffic. Traditional machine learning algorithms rely deeply on expert experience, and it is difficult for current traffic classification models to design effective handcraft features. Deep learning methods have been introduced in recent years to generate representations from raw network traffic data automatically. Previous deep learning-based malicious traffic detection methods generate representations from flow sequences or packet payload bytes. None of these methods simultaneously learn embeddings from flow sequence and packet payload bytes. Thus, we propose a novel ensemble model to draw fine-grained and multi-angle traffic representations for RAT traffic. The model extract (1) temporal features with convolution neural network (CNN) and the Reproducing Kernel Hilbert Space (RKHS) embedding method to model network flow sequence, (2) spatial features with autoencoder and bidirectional gated recurrent unit (Bi-GRU) network to model packet payload bytes, and (3) some stage-based attributes to enhance the identification ability of RAT traffic behaviors. According to the experimental result, our approach achieves better performance than previous works with a precision rate of 97.0% and a recall rate of 96.5%.