Verification of Neural Networks for Safety Critical Applications

Abstract

In recent years, Neural Networks (NNs) have been widely adopted in engineering automated driving systems with examples in perception, decision-making, or even end-to-end scenarios. As these systems are safety-critical in nature, they are too complex and hard to verify. For using neural networks in safety-critical domains, it is important to know if a decision made by a neural network is supported by prior similarities in the training process. Verifying a trained neural network is to measure the extent of the decisions made by the neural network, which are based on prior similarities in the training process. In this paper, we propose a runtime monitoring that can measure the reliability of the neural network trained to classify a new input based on prior similarities in the training set. In the training process, the runtime monitor stores the values of the neurons of certain layers, which represent the neurons activation pattern for each example in the training data. We use the Binary Decision Diagrams (BDDs) formal technique to store the neuron activation patterns in binary form. In the inference process, a classification decision measured by a hamming distance is made to any new input by examining if the runtime monitor contains a similar neurons activation pattern. If the runtime monitor does not contain any similar activation pattern, it generates a warning that the decision is not based on prior similarities in the training data. Unlike previous work, we monitored more layers to allow for more neurons activation pattern of each input. We demonstrate our approach using the MNIST benchmark set. Our experimental results show that by adjusting the hamming distance, 75.63% of the misclassified labels are unseen activation patterns, which are not similar to any stored activation patterns from the training time.