GTP Benchmarks for Gradual Typing Performance

Abstract

Reproducible, rigorous experiments are key to effective computing research because they provide grounding and a way to measure progress. Gradual typing is an emerging area that desperately needs such grounding. A gradual language lets programmers add types to part of a codebase while leaving the rest untyped. The critical research question is how to balance the guarantees that types provide against the run-time cost of enforcing them. Either weaker guarantees or better implementation methods could lead to answers, but without benchmarks for reproducibility there is no sound way to evaluate competing designs. The GTP Benchmark Suite is a rigorous testbed for gradual typing that supports reproducible experiments. Starting from a core suite of 21 programs drawn from a variety of applications, it enables the systematic exploration of over 40K gradually-typed program configurations via software for managing experiments and for analyzing results. Language designers have used the benchmarks to evaluate implementation strategies in at least seven major efforts since 2014. Furthermore, the benchmarks have proven useful for broader topics in gradual typing.