MTCR-AE: A Multiscale Temporal Convolutional Recurrent Autoencoder for unsupervised malicious network traffic detection

Abstract

The increasing sophistication of network attacks, particularly zero-day threats, underscores the need for robust, unsupervised detection methods. These attacks can flood networks with malicious traffic, overwhelm resources, or render services unavailable to legitimate users. Existing machine learning methods for zero-day attack detection typically rely on statistical features of network traffic, such as packet sizes and inter-arrival times. However, traditional approaches that depend on manually labeled data and linear structures often struggle to capture the intricate spatiotemporal correlations crucial for detecting unknown attacks. This paper introduces the Multiscale Temporal Convolutional Recurrent Autoencoder (MTCR-AE), an innovative framework designed to detect malicious network traffic by leveraging Multiscale Temporal Convolutional Networks (TCN) and Gated Recurrent Units (GRU). The MTCR-AE model captures both short- and long-range spatiotemporal dependencies while incorporating a temporal attention mechanism to dynamically prioritize critical features. The MTCR-AE operates in an unsupervised manner, eliminating the need for manual data labeling and enhancing its scalability for real-world applications. Experimental evaluations conducted on four benchmark datasets — ISCX-IDS-2012, USTC-TFC-2016, CIRA-CIC-DoHBrw2020, and CICIoT2023 — demonstrate the model’s superior performance, achieving an accuracy of 99.69%, precision of 99.63%, recall of 99.69%, and an F1-score of 99.66%. The results highlight the model’s capability to deliver state-of-the-art detection performance while maintaining low false positive and false negative rates, offering a scalable and reliable solution for dynamic network environments.