ExcePy: A Python Benchmark for Bugs with Python Built-in Types

As bugs of Python built-in types can cause code crashes, detecting them is critical to the robustness of the software. Researchers have concluded plenty of patterns for the bug causes and applied these patterns in detection tools. But these tools are only evaluated on handcrafted bugs or bugs obtained from QA pages. Because such bugs cannot reflect the complex code structures and various bug types encountered in real-world projects, the evaluation result is untrustworthy when applied to these projects. As a result, a collection of real-world reproducible bugs is essential for tool evaluation and future bug-related research. In this paper, we propose ExcePy, a benchmark for providing bugs of Python built-in types. We collect 180 bugs from the evolution of 15 real-world open-source Python projects on GitHub and then manually build test scripts for bug reproduction. Meanwhile, to improve tool evaluation efficiency, we present a code pruning strategy that can minimize buggy code size while retaining bug reproducibility and apply it to ExcePy to provide simplified buggy code. To demonstrate the benefits of ExcePy, we use three static analyzers and two fuzzers to detect bugs collected in ExcePy. We found that simplified code can significantly reduce running time and avoid many tool crashes, and bugs supplied by ExcePy can reveal limitations of existing tools in reporting real-world bugs.