METC: A Hybrid Deep Learning Framework for Cross-Network Encrypted DNS over HTTPS Traffic Detection and Tunnel Identification

Abstract

With the widespread adoption of DNS over HTTPS (DoH), network privacy and security have significantly improved, but detecting encrypted DoH traffic remains challenging, especially in heterogeneous environments. Existing research primarily focuses on desktops, neglecting mobile-specific detection.

To address this gap, we propose METC, a multi-stage hybrid learning framework for encrypted DoH traffic detection. We develop a mobile traffic collection tool supporting IPv6 and real-time inference and release the first mobile DoH dataset, comprising 38.21 GB of data.

METC integrates Convolutional Neural Networks (CNNs), Bidirectional Gated Recurrent Units (BiGRUs), and multi-head attention mechanisms, effectively capturing local traffic patterns, temporal dependencies, and key features to enhance cross-network generalization. Our CNN-BiGRU-Attention model achieves an F1-score of 97.34% in mobile DoH detection and 99.96%, 95.99%, and 94.65% in DoH-based tunnel traffic identification across three datasets. Additionally, it accurately identifies 10 tunneling tools, outperforming XGBoost in cross-network scenarios.

In summary, METC offers an innovative and efficient solution for encrypted DoH traffic detection and tunnel identification, advancing deep learning applications in network security.