Disclosing the Fragility Problem of Virtual Safety Testing for Autonomous Driving Systems

Abstract

In order to overcome the insufficiency of road testing in cost and coverage, vendors have been using 3D simulators to conduct virtual safety testing (VST) for autonomous driving. In this work, we report our observations of the unexpected issues in virtual safety testing: 1. Some commonly overlooked factors in simulator usage can subtly affect the running simulation, leading to false positive collision cases; 2. Minor changes in the simulation world can lead to drastically different testing results. We collectively refer to these unexpected or uncertain issues as the VST fragility problem. With the developed concrete cases, we explain the fragility problem in details and reason for the root causes accordingly. Moreover, we propose mitigation guidance based on the underlying causes of the fragility points.