An industrial case study of structural testing applied to safety-critical embedded software

Abstract

Effective testing of safety-critical real-time embedded software is difficult and expensive. Many companies are hesitant about the cost of formalized criteria-based testing and are not convinced of the benefits. This paper presents the results of an industrial case study that compared the normal testing at a company (manual functional testing) with testing based on the logic-based criterion of correlated active clause coverage (CACC). The evaluation was performed during the testing of embedded, real-time control software that has been deployed in a safety-critical application in the transportation industry. We found in our study that the test cases generated to satisfy the CACC criterion detected major safety-critical faults that were not detected by functional testing. We also found that the cost required for CACC testing was not necessarily higher than the cost of functional testing. There were also several faults that were found by the functional tests that were not found by CACC tests.