Synthesis of Benchmarks for the C Programming Language by Mining Software Repositories

Abstract

Compilers are usually distributed with a test framework. This framework supports the task of tuning optimizations and static analyses. As an example, clang has a test suite that, in March 2019, counted 259 benchmarks. Although in principle a large collection, this number is small once we consider the needs of the automatic tuning techniques that became fashionable recently. To mitigate the problems caused by such lack of benchmarks, this paper introduces a technique that allows the automatic construction of compilable programs out of open-source repositories. Our approach has made it possible to build, in less than 24 hours, a collection with over 500 thousand functions that clang can compile. In this paper, we show that such abundance of data gives us precise information about the behavior of compiler optimizations, and lets us create accurate prediction models. This collection of benchmarks is today freely available to the open-source community.