RaceBench: A Triggerable and Observable Concurrency Bug Benchmark

Concurrency bugs are one of the most harmful and hard-to-address issues in multithreaded software. Such bugs are hard to discover, reproduce, diagnose or fix due to their non-deterministic nature. Although more and more bug discovery solutions are proposed in recent years, it is difficult to evaluate them with existing concurrency bug datasets. The demand for building a high-quality benchmark of concurrency bugs emerges. In this paper, we present an automated bug injection solution to automatically inject representative concurrency bugs into real world multithreaded C/C++ programs, and present the first triggerable and observable concurrency bug benchmark RaceBench. We have conducted a large-scale empirical study on concurrency bugs, learned their patterns, and built a program state model to characterize them, which enables us to inject representative bugs. To make the bugs triggerable, we follow the dynamic execution traces of target programs and inject bugs at locations that are reachable from the program entry. To make the bugs observable, these bugs are followed by explicit security assertions, removing the requirement of sophisticated sanitizers to detect the existence of such bugs. We built a benchmark consisting of 1500 bugs injected into 15 programs, and evaluated four concurrency bug discovery tools and one general bug discovery tool with it. Results showed that existing concurrency bug discovery solutions are still in the early stage, and our benchmark could shed light on the future direction of improvements.