A Systematic Literature Review on Automated Software Vulnerability Detection Using Machine Learning

Abstract

In recent years, numerous Machine Learning (ML) models, including Deep Learning (DL) and classic ML models, have been developed to detect software vulnerabilities. However, there is a notable lack of comprehensive and systematic surveys that summarize, classify, and analyze the applications of these ML models in software vulnerability detection. This absence may lead to critical research areas being overlooked or under-represented, resulting in a skewed understanding of the current state of the art in software vulnerability detection. To close this gap, we propose a comprehensive and systematic literature review that characterizes the different properties of ML-based software vulnerability detection systems using six major research questions (RQs). Using a custom web scraper, our systematic approach involves extracting a set of studies from four widely used online digital libraries—ACM Digital Library, IEEEXplore, ScienceDirect, and Google Scholar. We manually analyzed the extracted studies to filter out irrelevant work unrelated to software vulnerability detection, followed by creating taxonomies and addressing research questions. Our analysis indicates a significant upward trend in applying ML techniques for software vulnerability detection over the past few years, with many studies published in recent years. Prominent conference venues include the International Conference on Software Engineering (ICSE), the International Symposium on Software Reliability Engineering (ISSRE), The Mining Software Repositories (MSR) conference, and the ACM International Conference on the Foundations of Software Engineering (FSE), while the Information and Software Technology (IST), the Computers & Security (C&S), and the Journal of Systems and Software (JSS) are the leading journal venues. Our results reveal that 39.1% of the subject studies use hybrid sources while 37.6% of the subject studies utilize benchmark data for software vulnerability detection. Code-based data are the most commonly used data type among subject studies, with source code being the predominant subtype. Graph-based and token-based input representations are the most popular techniques, accounting for 57.2% and 24.6% of the subject studies, respectively. Among the input embedding techniques, graph embedding and token vector embedding are the most frequently used techniques accounting for 32.6% and 29.7% of the subject studies. Additionally, 88.4% of the subject studies use DL models, with Recurrent Neural Networks (RNNs) and Graph Neural Networks (GNNs) being the most popular subcategories, while only 7.2% use classic ML models. Among the vulnerability types covered by the subject studies, CWE-119, CWE-20, and CWE-190 are the most frequent ones. In terms of tools used for software vulnerability detection, Keras with TensorFlow backend and PyTorch libraries are the most frequently used model-building tools accounting for 42 studies for each. Also, Joern is the most popular tool used for code representation accounting for 24 studies. Finally, we summarize the challenges and future directions in the context of software vulnerability detection, providing valuable insights for researchers and practitioners in the field.