CheckShake: Passively Detecting Anomaly in Wi-Fi Security Handshake using Gradient Boosting based Ensemble Learning

Recently, a number of attacks have been demonstrated (like key reinstallation attack, called KRACK) on WPA2 protocol suite in Wi-Fi WLAN, for which a patching is often challenging. In this article, we design and implement a system, called CheckShake, to passively detect anomalies in the handshake of Wi-Fi security protocols, in particular WPA2, between a client and an AP using COTS radios. Our proposed system works without decrypting any traffic and sniffing on multiple channels in parallel. It uses a state machine model for grouping Wi-Fi handshake packets and then perform deep packet inspection to identify the symptoms of the anomaly in specific stages of a handshake session. Our implementation of CheckShake does not require any modification to the firmware of the client or the AP or the COTS devices, it only requires to be physically placed within the range of the AP and its clients. We use both the publicly available dataset and our own data set for performance analysis of CheckShake. Using gradient boosting-based supervised machine learning (ML) models, we show that an accuracy around 98.50% with no false positive can be achieved using CheckShake in open sourced data that has non-zero probability of missing packets per group of packets.