Semantic-aware testing for object detection systems

Context:

Deep Learning-based object detection (OD) module is rapidly being the common basis for many popular autonomous systems such as self-driving cars and drones. Comprehensive robust testing is essential before deploying an OD module in safety-critical applications.

Objective:

We aim to address the following limitations of existing OD testing works: (1) they focus primarily on 2D OD with single-camera inputs rather than 3D OD with multi-camera fusion; (2) they rely on limited environmental changes or GAN transformations that inadequately cover diverse and complex real-world input; and (3) existing testing metrics remain unevaluated in the OD setting.

Methods:

We propose and develop a systematic semantic-aware testing framework named SeaT-OD capable of testing practical 3D OD systems based on fused image input by tackling several key technical challenges. Our approach introduces: (1) novel semantic-aware metrics defined over deep feature spaces applicable across diverse OD models; (2) a test generation algorithm using deep semantic transformation to enhance input semantic coverage; and (3) metric-guided test case selection for efficient model robustness improvement through targeted retraining.

Result:

We evaluated SeaT-OD on state-of-the-art commonly adopted 2D and 3D OD models based on fused image input in popular datasets from autonomous driving. Extensive experimental results show that existing OD testing works are insufficient, and SeaT-OD is effective in measuring the adequacy of testing practical 3D OD systems, generating high-quality test cases, and selecting test cases meaningful for improving the system robustness.

Conclusion:

Based on the results, we emphasize the importance of testing OD systems. Additionally, we present several observations that can direct future research and developments.