Comprehensively evaluating the perception systems of autonomous vehicles against hazards

Perception systems are vital for the safety of autonomous driving. In complex autonomous driving scenarios, autonomous vehicles must overcome various natural hazards, such as heavy rain or raindrops on the camera lens. Therefore, it is essential to conduct comprehensive testing of the perception systems in autonomous vehicles against these hazards, as demanded by the regulatory agencies of many countries for human drivers. Since there are many hazard scenarios, each of which has multiple configurable parameters, the challenges are (1) how do we systematically and adequately test an autonomous vehicle against these hazard scenarios, with measurable outcome; and (2) how do we efficiently explore the huge search space to identify scenarios that would induce failure?

In this work, we propose a Hazards Generation and Testing framework (HazGT) to generate a customizable and comprehensive repository of hazard scenarios for evaluating the perception system of autonomous vehicles. HazGT not only allows us to measure how comprehensively an autonomous vehicle (AV) has been tested against different hazards but also supports the identification of important hazards through optimization. HazGT supports a total of 70 kinds of hazards relevant to the visual perception of AVs, which are based on industrial regulations. HazGT automatically optimizes the parameter values to efficiently achieve different testing objectives. We have implemented HazGT based on two popular 3D engines, i.e., Unity and Unreal Engine. For the two mainstream perception models (i.e., YOLO and Faster RCNN), we have evaluated their performance against each hazard through extensive experiments, and the results show that both systems have much room to improve. In addition, our experiments also found that ChatGPT4 performs slightly worse than YOLO. Our optimization-based testing system is effective in finding perceptual errors in the perception models. The hazard images generated by HazGT are instrumental for improving perception models.