Detecting duplicate vulnerability records across databases

Vulnerability databases are critical repositories that aggregate information about known security vulnerabilities across various software products. However, the existence of multiple, heterogeneous databases often leads to duplicate vulnerability records, necessitating significant manual effort by maintainers to identify and consolidate these duplicates. This study addresses the challenge of detecting duplicate vulnerabilities across different databases by proposing a combined method that integrates cosine similarity measures with a fine-tuned BERT-based language model. We constructed a comprehensive duplicate vulnerability dataset by analyzing records from prominent databases such as CVE, OSV, and the GitHub Advisory Database. Our method was evaluated against several baseline techniques, including similarity-based and deep learning-based approaches, demonstrating superior performance across multiple metrics, including Hit Rate@N, Mean Reciprocal Rank (MRR), Mean Rank, and Median Rank. Additionally, our method proved effective in practical scenarios involving ongoing database maintenance, showcasing its ability to generalize to unseen data. The findings highlight the potential of integrating traditional similarity measures with advanced language models to enhance the accuracy and efficiency of duplicate vulnerability detection, thereby facilitating more reliable vulnerability management.