Hybrid machine learning approach for anomaly detection

Abstract

This research aims to improve anomaly detection performance by developing two variants of hybrid models combining supervised and unsupervised machine learning techniques. Supervised models cannot detect new or unseen types of anomaly. Hence in variant 1, a supervised model that detects normal samples is followed by an unsupervised learning model to screen anomaly. The unsupervised model is weak in differentiating between noise and fraud. Hence in variant 2, the hybrid model incorporates an unsupervised model that detects anomaly is followed by a supervised model to validate an anomaly. Three different datasets are used for model evaluation. The experiment is begun with 5 supervised models and 3 unsupervised models. After performance evaluation, 2 supervised models with the highest F1-Score and one unsupervised model with the best recall value are selected for hybrid model development. The variant 1 hybrid model recorded the best recall value across all the experiments, indicating that it is the best at detecting actual fraud and less likely to miss it compared to other models. The variant 2 hybrid model can improve the precision score significantly compared to the original unsupervised model, indicating that it is better in separating noise from fraud,