nNFST: A single-model approach for multiclass novelty detection in network intrusion detection systems

Abstract

The rapid evolution of cyberattack techniques necessitates advanced intrusion detection systems (IDS) capable of multiclass novelty detection (MND), accurately classifying known attacks while identifying novel ones. Despite numerous successful studies focused on multi-class attack classification or novel attack detection separately, a significant research gap remains in achieving the effective MND for IDS. In this paper, we introduce the neighbour null Foley–Sammon transformation (nNFST), a novel single-model algorithm designed to address the MND challenge in IDS. nNFST employs a novel technique based on the inverse nearest neighbour algorithm to compute within-class and between-class variation. This technique preserves both the local distribution structure within each class and the global distribution structure across classes, thereby mitigating the impact of singular points on the algorithm and enhancing accuracy on complex data. Furthermore, nNFST leverages the kernel trick to improve detection accuracy and sparse matrix multiplication to reduce training costs. Comprehensive evaluation results on four public datasets demonstrate nNFST’s superior performance compared to related works in different tasks, achieving 97.12% to 99.56% accuracy in multiclass classification tasks, 94.53% to 99.33% accuracy in novel attack detection tasks, and a 0.825 to 0.975 Matthews correlation coefficient in MND tasks. These results highlight nNFST’s potential to significantly enhance IDS capabilities by concurrently classifying known attacks and identifying unknown attacks.