On the Use of Open-Source C/C++ Static Analysis Tools in Large Projects

Abstract

Software applications are frequently deployed with security vulnerabilities that may open the door to attacks. In business-critical scenarios, such attacks may lead to significant financial and reputation losses. Static Analysis Tools (SATs), which analyze the source code without executing it, can be used to detect potential faults in the source code, including security vulnerabilities. However, many false alarms are normally reported, leading teams to discard the use of such tools, especially on large software projects. Existing works have dealt with the evaluation of SATs, but they are mostly based on small pieces of code designed to support the evaluation. In this paper, we present and discuss the results of the execution of two Open-Source C/C++ SATs (CPPCheck and Flawfinder) on the large open-source project Mozilla. Our goal is to study the applicability of SATs in a large project and the vulnerability categories they can detect. Results show that CppCheck could detect 83.5% of the vulnerabilities, and Flawfinder could detect 36.2%, although the number of false alarms is high (7.2% for CppCheck and 93.2% for Flawfinder). Regarding the different categories, the two SATs showed quite diverse performances (e.g., CppCheck was able to detect $92.6% of Data Protection vulnerabilities and 62.5% of Coding Practices vulnerabilities, while false alarms were 99.1% and 99.9%, respectively).