On using a benchmark to evaluate C++ extractors

In this paper, we take the concept of benchmarking, as used extensively in computing, and apply it to the evaluation of C++ fact extractors. We demonstrate the efficacy of this approach by developing a prototype benchmark, CppETS 1.0 (C++ Extractor Test Suite, pronounced 'see-pets') and collecting feedback in a workshop setting. The CppETS benchmark characterises C++ extractors along two dimensions: accuracy and robustness. It consists of a series of test buckets that contain small C++ programs and related questions that pose different challenges to the extractors. As with other research areas, benchmarks are best developed through technical work and consultation with a community, so we invited researchers to apply CppETS to their extractors and report on their results in a workshop. Four teams participated in this effort, evaluating the four extractors Ccia, cppx, the Rigi C++ parser and TkSee/SN. They found that CppETS gave results that were consistent with their experience with these tools and therefore had good external validity. Workshop participants agreed that CppETS was an important contribution to fact extractor development and testing. Further efforts to make CppETS a widely-accepted benchmark will involve technical improvements and collaboration with the broader community.