IFDRF: Advancing Anomaly Detection with a Hybrid Machine Learning Model

Abstract

Anomaly detection is the identification of aberrations in the dataset using statistical methods or machine learning algorithms. It is widely performed using unsupervised learning algorithms because labelling the data manually can be expensive. While unsupervised anomaly detection is sufficient for data cleaning, this is not the case in real-world applications, where accuracy is of the utmost importance. For example, it would be unacceptable to misdiagnose someone as not having breast cancer and not provide them with treatment because our model failed to recognize it as an anomaly. In this paper, we propose an optimized model—IFDRF (Isolation Forest, DBSCAN, and Random Forest) that has incorporated feedback (corrections) into the unsupervised detection model. IFDRF is a novel hybrid model combining an unsupervised learning model at the first layer followed by a clustering model at the second layer and a supervised learning model at the end. The proposed model tunes the unsupervised learning model followed by a model fitting with the help of the feedback mechanism. It obviates the need to label the entire dataset and thus increases the scope of anomaly detection applications. We have compared our proposed model to the existing state-of-the-art anomaly detection baseline models to show its efficacy. The proposed model performed significantly (\(P{\text{-value}} < 2.2 \times {{10}^{{ - 16}}}\)) better than the other algorithms, with an AUC score of 0.875.