Detection of Brute-Force Attacks in End-to-End Encrypted Network Traffic

Abstract

Network intrusion detection systems (NIDSs) can detect attacks in network traffic. However, the increasing ratio of encrypted connections on the Internet restricts their ability to observe such attacks. This paper proposes a completely passive method that allows to detect brute-force attacks in encrypted traffic without the need to decrypt it. For that, we propose five novel metrics for attack detection which quantify metadata like packet size or packet timing. We evaluate the performance of our method with synthetically generated but realistic traffic as well as on real-world traffic from a Tor exit node on the Internet. Our results indicate that the proposed metrics can reliably detect brute-force attacks in encrypted traffic in protocols like HTTPS, FTPS, IMAPS, SMTPS, and SSH. Simultaneously, our approach causes only a few false positives, achieving an F-measure between 75% and 100%.