Automatic Detection of Malicious URLs using Fine-Tuned Classification Model

Abstract

URLs are frequently used to surf the Internet in modern society. Especially after the outbreak of the COVID-19, quarantine makes needs for the Internet and usages of URLs reach an unprecedented level. Unfortunately, not every URL is believable, because some of them can attack your computer, steal your personal information, and even spread computer virus like Trojan. This project designs machine learning algorithms to detect malicious URLs efficiently and protect Internet users from malicious URLs. In this project, the goal is to get a fine-tuned machine learning model, I will first introduce the dataset of URLs which is crucial to train the model. Then I will show the procedure and some findings of my data exploration. After that, I will present the methods including the algorithms and some improvements I make to optimize my model. Finally, I will show the results and the conclusion. To make my models better, I not only apply hyperparameter tuning, but also data resampling and cross validation to the model. The procedure is repeated several times to ensure the stability. In order to evaluate the performance of my models accurately, I adopt multiple methods. After improving the algorithms, by using the F1 score to evaluate performance, the result boosts significantly from original 0.14 to around 0.90. With the ultimate well-trained model, we can predict the safety of all the URLs on the Internet accurately, which can secure our personal information and data.