New Version, New Answer: Investigating Cybersecurity Static-Analysis Tool Findings

Abstract

Automated detection of vulnerabilities and weaknesses in binary code is a critical need at the frontier of cybersecurity research. Cybersecurity static-analysis tools aim to detect and enumerate vulnerabilities and weaknesses. Two popular tools are CVE Binary Tool (cve-bin-tool) and cwe-checker. Cve-bin-tool reports vulnerabilities using Common Vulnerabilities and Exposures (CVE) whereas cwe-checker reports weaknesses using Common Weakness Enumeration (CWE). Despite widespread use, the consistency with which these tools report vulnerabilities and weaknesses (herein, “findings”) was unaddressed. We conducted a systematic investigation of 660 unique binaries taken from a Kali Linux distribution, evaluated each binary with multiple versions of the static-analysis tools, and investigated how the findings changed according to the version of the static-analysis tool used. We expected some variation in findings commensurate with the software-development life cycle. However, the number and magnitude of the changes in findings reported across versions were substantial. New versions gave new answers.