Fast and Accurate Machine Learning-based Malware Detection via RC4 Ciphertext Analysis*

Abstract

Recent malware increases its viability by employing ciphers which help to hide malicious intention and/or behavior against detection schemes. So far, many efforts have been made to detect malware and to prevent it from damaging clients by monitoring network packets. However, these conventional detection schemes tend to treat an encrypted packet as legitimate due to the hardness of extracting information from ciphertexts. Cryptoanalysis of each packet flowing over a network might be one feasible solution to the problem. However, this approach is computationally expensive and lacks accuracy, and thus it is consequently not a practical solution. To address the problem, we firstly introduce a discovery that a fixed encryption key generates unique statistical patterns on RC4 ciphertexts. To the best of our knowledge, this unique signature has never been discussed in the literature. Then, we propose a machine learning-based detection scheme that can identify malware packets efficiently and accurately by leveraging the discovery. The proposed scheme directly analyze network packets without decrypting ciphertexts. Moreover, our analysis demonstrates the proposed scheme requires only a tiny subset of the network packet.