Systematic concurrency testing using CHESS

Abstract

Concurrency testing should aim for systematic coverage of thread interleavings. The most common method used today is stress testing, where the program is run under load with lots of threads. While this indirectly increases the variety of thread interleavings, the coverage is neither sufficient and nor predictable---stories are legend of the so-called "Heisenbugs" that rarely surface during testing and are very hard to debug. In this talk, I will argue for a new notion of concurrency testing called scenario testing and describe CHESS, a tool we have developed towards that end. A user of CHESS provides simple concurrency scenarios and CHESS systematically enumerates all thread interleavings of these scenarios. CHESS employs model checking techniques to effectively focus on potentially bug-yielding schedules and provides sound quantifiable notions of coverage. On finding an error, CHESS has the capability to replay the erroneous interleaving, greatly simplifying the debugging process. CHESS has been successfully integrated with several codebases inside Microsoft and is used daily by the test teams. CHESS has found numerous bugs in systems that have been stress-tested for months. Additionally, CHESS has successfully reproduced many stress-test crashes that were previously hard to debug. The latter shows that many bugs that are found in stress-testing can indeed be reproduced in simple scenarios, a good validation for scenario testing. CHESS is available for download at http://research.microsoft.com/CHESS. The talk will contain enough material to act as a tutorial for first-time users. I will also describe the challenges in applying CHESS to a new codebase and how to address them.