A Comparative Study of Automatic Program Repair Techniques for Security Vulnerabilities

Abstract

In the past years, research on automatic program repair (APR), in particular on test-suite-based approaches, has significantly attracted the attention of researchers. Despite the advances in the field, it remains unclear how these techniques fare in the context of security—most approaches are evaluated using benchmarks of bugs that do not (only) contain security vulnerabilities. In this paper, we present our observations using 10 state-of-the-art test-suite-based automatic program repair tools on the DARPA Cyber Grand Challenge benchmark of vulnerabilities in C/C++. Our intention is to have a better understanding of the current state of automatic program repair tools when addressing security issues. In particular, our study is guided by the hypothesis that the efficiency of repair tools may not generalize to security vulnerabilities. We found that the 10 analyzed tools can only fix 30 out of 55 vulnerable programs—54.6 % of the considered issues. In particular, we found that APR tools with atomic change operators and brute-force search strategy (AE and GenProg) and brute-force functionality deletion (Kali) overall perform better at repairing security vulnerabilities (considering both efficiency and effectiveness). AE is the tool that individually repairs most programs with 20 out of 55 programs (36.4%). The causes for failing to repair are discussed in the paper, which can help repair tool designers to improve their techniques and tools.