Defining metrics for scenario-based evaluation of autonomous vehicle models

Abstract

The paper deals with the evaluation of autonomous vehicles along with the quantification of their behavior and maneuvers. The article outlines the positive aspects of autonomy and lists several arguments in their favor, e.g. convenience and efficiency considerations. Furthermore, it also addresses the associated difficulties including the feasibility of road testing and the establishment of appropriate simulations. The current work aims to define methods providing objective indicators to compare algorithms solving the complex tasks of road transport. Rule-based, supervised and reinforcement learning control models, test environments, accelerated test methods and assessment indicators of the corresponding literature are reviewed and evaluated. After investigating the different metrics, we formulate an evaluation framework that can be applied in the development and assessment process of new artificial intelligence controlled models. As an outcome of this work, we aim to aid a missing sector in the field of autonomous driving function development by collecting and defining metrics that intend to help qualitatively evaluate and compare algorithms. The key aspect during the definition of the suggested method was to ensure its extensive applicability by selecting only metrics that can be obtained from the already installed sensors of the vehicles. Additionally, we also assess multiple agents to observe how their behavior compares and whether the proposed metrics reflect the expected behavior.