Can generative AI detect and fix real-world cryptographic misuses?

Abstract

We evaluate ChatGPT’s ability to detect and fix cryptographic API misuses. We show that GPT-4o can achieve F1 scores above 0.90 on two established benchmarks. We also assess the model on real-world code: on the GitHub samples, it attains an F1 score of 0.84 and a Matthews Correlation Coefficient (MCC) of 0.81; on the Android samples, it achieves an F1 score of 0.85 but a slightly lower MCC of 0.76. We noted that several factors such as naming conventions and code structure influence GPT’s performance. However, when it correctly flags a misuse, it is often able to suggest effective fixes. We also reported the identified misuses in GitHub repositories and received promising feedback from developers. Finally, a comparison between GPT and a state-of-the-art crypto-misuse detector shows GPT’s strong potential for adoption in real-world settings.