MiMIs: Simple, Efficient, and Fast Bounded-Exhaustive Test Case Generators

Abstract

Bounded-exhaustive test case generators are important bug-finding tools, epitomized with tools like Korat, UDITA, and SciFe. All such tools are practically limited by scale; a given testing problem may be too large to generate all tests within reasonable time, or prohibitive amounts of memory may be required for this generation. The tools themselves may have lengthy implementations which hide their own bugs, or they may require users to familiarize themselves with many operations. In this paper, we strive to push the boundaries of what test case generators can scale to, while preserving a simple implementation and user interface. To this end, we introduce a novel programming abstraction: Memoized Monadic Iterators (MiMIs). MiMIs combine the disparate ideas of ASTGen’s iterators, LogicT’s additive monad interface, and SciFe’s memoization into a single coherent programming abstraction. Our evaluation shows that MiMIs are typically faster than the state of the art with orders of magnitude less memory used, and that MiMIs overall scale better to big testing problems than any competitor. Thanks to their fundamental design, MiMIs are implemented in under 200 LOC (compared to thousands with most competitors), and require users to familiarize themselves with collectively only 7 operations and types (less than half as many as the state of the art), all without sacrificing generator conciseness.