EMTD: Efficient encrypted malware traffic detection based on adaptive meta-path guided graph propagation

Abstract

Given the growing challenges posed by encrypted Android malware, developing effective detection methods is crucial to understanding the evolution of malware families and designing preventive security measures. Existing detection methods for encrypted malware traffic primarily focus on extracting features at the single-flow level and multi-flow context level, failing to capture the evolutionary associations within known malware families and their previously unseen variants, which compromises detection effectiveness. Graph-based modeling methods have advantages in expressing traffic association features, but scalability issues present new challenges for detection timeliness. To address these limitations, we propose a novel two-stage detection framework named Encrypted Malware Traffic Detection (EMTD). In the first phase, our method MFGDect models encrypted traffic as a heterogeneous information network and applies a multilayer heterogeneous attention mechanism to learn semantic associations among traffic flows. This enables adaptive family-aware representation and improves detection performance under encryption. Furthermore, we design MFGDect++, an extension of our base model that introduces adaptive meta-path guided graph propagation, enabling efficient incremental detection of new traffic samples without re-graphing or model retraining. This mechanism significantly reduces the average detection time to 135 ms per sample, demonstrating strong scalability. Experiments on public datasets demonstrate that EMTD outperforms existing baselines, achieving an average improvement of 9.62% in malicious sample recall and a 2.32% increase in F1 score, while maintaining low resource overheads and strong adaptability to large-scale graph data.