Evaluation of Neural Network Verification Methods for Air-to-Air Collision Avoidance

Neural network approximations have become attractive to compress data for automation and autonomy algorithms for use on storage-limited and processing-limited aerospace hardware. However, unless these neural network approximations can be exhaustively verified to be safe, they cannot be certified for use on aircraft. An example of such systems is the unmanned Airbone Collision Avoidance System (ACAS Xu), which is a very popular benchmark for open-loop neural network control system verification tools. This paper proposes a new closed loop extension of this benchmark, which consists of a set of ten closed loop properties selected to evaluate the safety of an ownship aircraft in the presence of a co-altitude intruder aircraft. These closed loop safety properties are used to evaluate 5 of the 45 neural networks that comprise the ACAS Xu benchmark (corresponding to co-altitude cases) as well as the switching logic between the 5 neural networks. The combination of nonlinear dynamics and switching between five neural networks is a challenging verification task accomplished with star set reachability methods in two verification tools. The safety of the ownship aircraft under initial position uncertainty is guaranteed in every scenario proposed. using the NNV and nnenum tools on 5 of 45 ACAS Xu neural networks with switching between five to co-altitude collision cases. Both tools show reachable states of an ownship with dynamics on a collision course with an intruder The reachability computation and ownship by 5 switched NNs. The verification switch collision free even a large initial This study showed that do a comprehensive safety verification analysis of a complex air collision advisory system for but further research determine flights In the tool showed and computation