Characterizing the 'security vulnerability likelihood' of software functions

Software maintainers and auditors would benefit from a tool to help them focus their attention on functions that are likely to be the source of security vulnerabilities. However, the existence of such a tool is predicated on the ability to characterize a function's 'security vulnerability likelihood'. Our hypothesis is that functions near a source of input are most likely to contain security vulnerability. These functions should be a small percentage of the total number of functions in the system. To validate this hypothesis, we performed an experiment involving thirty one vulnerabilities in thirty open source systems. This paper describes the experiment, its outcome, and the tools used to conduct it. It also describes the FLF (front line functions) finder, which is a tool that was developed using knowledge gathered from the outcome of the experiment. This tool automates the detection of high-risk functions. To demonstrate the effectiveness of the FLF finder, three open source applications with known vulnerabilities were tested. In addition to this test, a case study was performed on the privilege separation code in the OpenSSH server daemon.