Discovering Decision Manifolds to Assure Trusted Autonomous Systems

Abstract

Developing and fielding complex systems requires proof that they are reliably correct with respect to their design and operating requirements. Especially for autonomous systems, which exhibit unanticipated emergent behavior, fully enumerating the range of possible correct and incorrect behaviors is intractable. Therefore, we propose an optimization-based search technique for generating high-quality, high-variance, and nontrivial data, which captures the range of correct and incorrect responses a system could exhibit. This manifold between desired and undesired behavior provides a more detailed understanding of system reliability than traditional testing or Monte Carlo simulations. After discovering data points along the manifold, we apply machine learning techniques to quantify the decision manifold's underlying mathematical function. Such models serve as correctness properties, which can be utilized to enable both verification during development and testing, as well as continuous assurance during operation, even amidst system adaptations and dynamic operating environments. This method can be applied in combination with a simulator in order to provide evidence of dependability to system designers and users, with the ultimate aim of establishing trust in the deployment of complex systems. In this proof-of-concept, we apply our method to a software-in-the-loop evaluation of an autonomous vehicle.