Monitoring runtime input data distribution for the safety of the intended functionality in perception systems

Abstract

Safe and reliable environmental perception is essential for the highly automated or even autonomous operation of agriculture machines. However, developing a functionally safe and reliable AI-powered perception system is challenging, especially in safety-critical applications, due to the nature of AI technologies. This article is motivated by the need to constrain an AI-powered perception system to work within a predefined safe envelope, ensuring that the acceptable behaviour of AI technology is maintained. The acceptable behaviour of AI technology is assessed based on the distribution of its training data. However, verifying the model’s performance becomes challenging when it encounters unseen, out-of-distribution input data. This article proposes an image quality safety model (IQSM) that estimates the confidence in the safety of the intended functionality for a runtime input image within a perception system, even when faced with unseen out-of-distribution runtime input images. If the confidence level falls below the “minimum performance threshold” required for safe operation, the IQSM detects that the intended functionality is unsafe for performing highly automated operations. On a test set of 1,592 images comprising clear, dirty, foggy, raindrop-covered, and over-exposed, IQSM classified images as safe or unsafe with accuracies ranging from 97.6 % to 98.9 %. This demonstrates its ability to effectively detect acceptable runtime input images and ensure the acceptable behaviour of an intended function in world scenarios. The IQSM can prevent malfunctions in perception systems, such as failing to detect obstacles due to adverse weather conditions. It facilitates the integration of fail-safe architectures across various applications, including highly automated agricultural machinery, thereby contributing to the safety and reliability of the intended functionality.