Towards effective black-box attacks on DoH tunnel detection systems

The introduction of DNS-over-HTTPS (DoH) aims to mitigate the security vulnerabilities of traditional DNS. However, attackers have begun exploiting DoH to establish tunnels for malicious activities. Machine learning (ML)-based network intrusion detection systems (NIDSs) have emerged as a promising approach for detecting DoH tunnel attacks. Paradoxically, these ML models are susceptible to adversarial machine learning attacks. A growing number of researchers are investigating adversarial techniques to circumvent NIDS, yet they neglect the real-world viability of implementing these attack strategies under specific network constraints. To address this gap, we propose a black-box attack framework leveraging the transferability of adversarial samples, along with an adversarial sample generation algorithm called Strategic Feature-Adaptive Adversarial Attack (SFAA) which serves as the black-box attack framework’s core component. SFAA incorporates feature correlations and feature importance to optimize the perturbation direction, thereby generating more realistic adversarial samples. In the context of DoH intrusion attacks, we employ our proposed black-box attack framework to carry out adversarial attacks on commonly used and highly effective ML models. Our experimental results demonstrate that the proposed black-box attack framework effectively evades ML models, and adversarial samples generated by SFAA achieve an attack success rate (ASR) of 63.26%, surpassing state-of-the-art adversarial attacks, including the Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Projected Gradient Descent (PGD), DeepFool, Carlini & Wagner (C&W), and Jacobian Saliency Map Attack (JSMA). Moreover, we propose a defense framework combining adversarial training and confidence-driven secondary classification, providing a novel paradigm for the robust design of machine learning models to mitigate adversarial attacks.