Detection of Encrypted Malicious Network Traffic using Machine Learning

The proliferation of encrypted network traffic necessitates an innovative machine learning traffic analysis approach which does not rely on pattern matching or the payload content of the packets to detect malicious / suspicious communications. Encryption of Internet traffic has increasingly become a typical best practice, making network packet content analysis yield diminishing returns. A majority of internet traffic is now protected using the cryptographic protocol known as Transport Layer Security (TLS). Malware authors have also followed this trend with the use of TLS to hide malicious network communications. We propose a malicious communication detection mechanism using a Support Vector Machine (SVM) and an alternative with a Convolutional Neural Network (CNN). Both methods achieve respectable results and a low False Positive Rate (FPR). However, the SVM method outperforms the CNN method in all evaluation metrics presented. Lastly, we propose future work to experiment with transport layer size and direction as features and automate feature engineering by using raw packet traffic with a CNN augmented with a Long Short-term Memory (LSTM) for detection of malicious traffic.