System-level simulation-based verification of Autonomous Driving Systems with the VIVAS framework and CARLA simulator

Abstract

Ensuring the safety and reliability of increasingly complex Autonomous Driving Systems (ADS) poses significant challenges, particularly when these systems rely on AI components for perception and control. In the ESA-funded project VIVAS, we developed a comprehensive framework for system-level, simulation-based Verification and Validation (V&V) of autonomous systems. This framework integrates a simulation model of the system, an abstract model describing system behavior symbolically, and formal methods for scenario generation and verification of simulation executions. The automated scenario generation process is guided by diverse coverage criteria.

In this paper, we present the application of the VIVAS framework to ADS by integrating it with CARLA, a widely-used driving simulator, and its ScenarioRunner tool. This integration facilitates the creation of diverse and complex driving scenarios to validate different state-of-the-art AI-based ADS agents shared by the CARLA community through its Autonomous Driving Challenge. We detail the development of a symbolic ADS model and the formulation of a coverage criterion focused on the behaviors of vehicles surrounding the ADS. Using the VIVAS framework, we generate and execute various highway-driving scenarios, evaluating the capabilities of the AI components. The results demonstrate the effectiveness of VIVAS in automating scenario generation for different off-the-shelf AI solutions.