Efficient Critical Data Generation Framework for Vision Sensors of Autonomous Vehicle Perception System

Abstract

Visual sensors are essential for the perception systems of autonomous vehicles (AVs) and for ensuring driving safety. While data-driven perception methods perform well in common scenes, they often struggle with critical situations, leading to potential system failures and accidents. To overcome these, we propose a novel approach that fine-tunes large language models integrating a heuristic scene interpreter and employs inspired visual data generation techniques to produce data that closely mimics real-world conditions. This method is implemented on the device designed to inject the generated visual data directly into real sensors, enabling accurate performance assessments of AV perception systems. Authenticity, rationality, and quality of the generated scenes are evaluated through extensive experiments. Experimental results demonstrate that our method significantly enhances the generation of critical data and underscores the substantial value of our approach to improving the safety and reliability of AVs.