Covert timing channel detection based on isolated binary trees

Abstract

As a communication method for concealing information, the covert network channel is often exploited for malicious purposes due to its inherently difficult-to-detect nature, posing potential risks to network security. In this paper, we propose a detection method based on isolated binary trees, aiming to address the problem of the novel covert channel imitating legitimate traffic patterns and injecting additional anomalies to evade detection. This method is based on the Isolation Forest algorithm, which can be classified into different categories by analyzing the stepwise function features of network traffic and using isolation binary trees generated with random split thresholds. At the same time, we validate the proposed detection model using a publicly available dataset. The experimental results demonstrate that eliminating outliers significantly enhances the stepwise function features while preserving the original form of legitimate traffic. Compared to the model without outlier handling, the average AUC scores for TRCTC and Jitterbug improved by 7.37% and 2.23%, respectively. Furthermore, we achieved superior performance on a new channel named $ϵ$-$\kappa$libur and $ϵ$-$\kappa$libur-O compared to using deep learning-based detection methods.