Critical Configuration Testing for Autonomous Driving Systems---Targeting the Achilles' Heel of Four Open Autopilots

Abstract

The question of how to validate autonomous driving systems (ADS) has been the subject of much study and debate. The complexity and heterogeneity of ADS make testing, preferably by simulation, the main validation method. One of the difficulties of ADS testing lies in the fact that autopilot failures are rare events, and their exposure must be achieved by applying critical test cases that generate high-risk scenarios, whereas solely increasing criticality may lead to unavoidable failures that cannot reveal genuine defects. We propose the Critical Configuration Testing (CCTest) approach, which evaluates an ego vehicle driven by its autopilot under configurations involving conflicts between vehicles. High-risk scenarios are generated by applying test cases that specify the most critical initial kinematic attributes of the vehicles and worst-case movements of non-ego vehicles. This approach differs from existing methods in two key respects. First, it generates test cases that ensure the tested autopilot has a provably safe policy, so that any observed failure reflects a genuine defect in the autopilot. Second, it targets short scenarios involving interactions between at most three relevant vehicles, enabling precise identification of the most critical test cases. We present the mathematical framework for calculating critical parameter values for the test cases, as well as its application to four open autopilots: Apollo, Autoware, and the built-in autopilots in the Carla and LGSVL simulators. Our experiment reveals numerous failures in critical configurations, including accidents, traffic rule violations, blockages, and software failures, that would be difficult for existing methods to fully capture.