Local performance evaluation of AI-algorithms with the generalized spatial recall index

Abstract

Abstract We have developed a novel metric to gauge the performance of artificial intelligence (AI) or machine learning (ML) algorithms, called the Spatial Recall Index (SRI). The novelty is the spatial resolution of a standard performance indicator, as a Recall value is assigned to each individual pixel. This generates a distribution of the performance of a given AI-algorithm with the resolution of the images in the dataset. While the mathematical basis has already been presented before, here we demonstrate the usage on more datasets and delve into in-depth application examples. We examine both the MS COCO and the Berkeley Deep Drive datasets, using a state-of-the-art object detection algorithm. The dataset is degraded using a physical-realistic lens-model, where the optical performance varies over the field of view, as a real camera would. This study highlights the usefulness of the SRI, as every image has been taken by realistic optics. A generalization, the GSRI is introduced, from which we derive SRIA, weighting with object area and SRIrisk intended for autonomous driving. Finally, these metrics are compared.