Selective Classification of Sequential Data Using Inductive Conformal Prediction

Abstract

Cyber-Physical Systems (CPS) operate in dynamic and uncertain environments where the use of deep neural networks (DNN) for perception can be advantageous. However, DNN integration in CPS is not straightforward. Perception outputs must be complemented with assurance metrics that represent if they can be trusted or not. Further, the inputs to DNNs are typically sequential capturing time-correlated data that can affect the accuracy of the predictions since machine learning models require inputs to be independent and identically distributed. In this paper, we propose a selective classification approach that rejects predictions that are not trustworthy. We quantify the credibility and confidence of each prediction by computing aggregate p-values from multiple subsequent inputs. We examine different multiple hypothesis testing approaches for combining p-values computed using Inductive Conformal Prediction (ICP) focusing on their ability to produce valid p-values for sequential data. Empirical evaluation results using the German Traffic Sign Recognition Benchmark demonstrate that ICP validity can be recovered when p-values from sequential inputs are combined and selective classification based on aggregate p-values produces predictions with less risk.