Test and Evaluation Harnesses for Learning Systems

There is an increasing demand for operational uses of machine learning (ML), however, a lack of best practices for test and evaluation (T &E) of learning systems is a hindrance to supply. This manuscript proposes a new framework for best practices, described as T &E harnesses, that corresponds principally to the task of engineering a learning system-in contrast to the status quo task of solving a learning problem. The primary difference is a question of scope. This manuscript places T &E for ML into the broader scope of systems engineering processes. Importantly, two challenge problems, acquisition and operations, are used to motivate the use of T &E harnesses for learning systems. This manuscript draws from recent findings in experimental design for ML, combinatorial interaction testing of ML solutions, and the general systems modeling of ML. The concept of T &E harnesses is closely tied to existing models of systems engineering processes. We draw the conclusion that existing best practices for T &E form a subset of what is needed to rigorously test for system-level satisfaction of stakeholder needs.