Coverage metrics for verification of concurrent SystemC designs using mutation testing

Abstract

Design verification has grown to dominate the cost of electronic system design; however, designs continue to be released with latent bugs. A verification test suite developed for a sequential program is not adequate for a concurrent program. A major problem with design verification of concurrent systems is the lack of good coverage metrics. Coverage metrics are heuristic measures of the exhaustiveness of a test suite. High coverage, in general, implies fewer bugs. SystemC is the most popular concurrent system level modeling language used for designing SoCs in the industry. We propose to attack the verification quality problem for concurrent SystemC programs by developing novel mutation testing based coverage metrics. Mutation testing has successfully been applied in software testing and RTL designs. In this paper, we develop a comprehensive set of mutation operators for concurrency constructs in SystemC. Our approach is also unique in that we define a novel concurrent coverage metric considering multiple execution schedules that a concurrent program can generate. This metric allows us to adequately measure the coverage for concurrent programs. We performed experiments with various designs including a large industrial design and obtained favorable results on multiple applications.