Measuring the Gain of Automatic Debug

The purpose of regression testing is to quickly catch any deterioration in quality of a product under development. The more frequently tests are run, the earlier new issues can be detected resulting in a larger burden for the engineers who need to manually debug all test failures, many of which are failing due to the same underlying bug. However, there are software tools that automatically debug the test failures back to the faulty change and notifies the engineer who made this change. By analyzing data from a real commercial ASIC project we aimed to measure whether bugs are fixed faster when using automatic debug tools compared to manual debugging. All bugs in an ASIC development project were analyzed over a period of 3 months in order to determine the time it took the bug to be fixed and to compare the results from both automatic and manual debug. By measuring the time from when the bug report was sent out by the automatic debug tool until the bug was fixed, we can show that bugs are fixed 4 times faster with automatic debug enabled. Bug fixing time was on average 5.7 hours with automatic debug and 23.0 hours for manual debug. The result was achieved by comparing bugs that were automatically debugged to those issues that could not be debugged by the tool, because those issues were outside the defined scope of the device under test. Such issues are still reported by the automatic debug tool but marked as requiring manual debug and is consequently a good point of comparison. A 4 times quicker bug fixing process is significant and can ultimately contribute to a shortening of a development project as the bug turnaround time is one of the key aspects defining the length of a project, especially in the later phase just before release.