Towards Effective Sampling of SMT Solutions

Abstract

This work focuses on effectively generating diverse solutions for satisfiability modulo theories (SMT) formulas, targeting the theories of bit-vectors, arrays, and uninterpreted functions, which is a critical task in software and hardware testing. Generating diverse SMT solutions helps uncover faults and detect safety violations during verification and testing. This gives rise to the SMT sampling problem, whose goal is to construct a diverse solution set that achieves comprehensive coverage of the constraint space. While high coverage is crucial for exploring system behaviors, the number of generated solutions should also be kept small in software engineering scenarios. Since each SMT solution typically corresponds to a test case or test input, generating fewer solutions for the same coverage directly reduces testing time and resource consumption, thereby improving testing efficiency. In this work, we introduce PanSampler, a novel SMT sampler that achieves high coverage with a small number of solutions. PanSampler incorporates a new diversity-aware SMT algorithm and a post-sampling optimization strategy, which together enhance its practical performance. It iteratively samples solutions, evaluates candidates, and employs local search to refine solutions, ensuring high coverage with a small number of samples. Extensive experiments on practical benchmarks demonstrate that PanSampler exhibits a significantly stronger capability to reach high target coverage, while requiring fewer solutions than state-of-the-art samplers to achieve the same coverage level. Furthermore, our empirical evaluation on practical subjects, which are collected from real-world software systems, shows that PanSampler achieves stronger fault detection capability. Moreover, when reaching the same fault detection effectiveness as the competitors, PanSampler greatly reduces the number of required solutions, thereby improving testing efficiency. Our evaluations also validate the effectiveness of PanSampler’s core components. PanSampler represents a significant advancement in SMT sampling for software engineering, contributing to the cost reduction of software testing and hardware verification.