Assessing tactical alert function accuracy performance

An effective test program for the evaluation of the performance of the Short Term Conflict Alert (STCA) function must consider the definition of the test scenario, assessment metrics, and alert classification rules. This paper discusses test issues encountered using a realistic scenario based on live data. The paper describes a method to obtain systematic and automated measurements of nuisance rate, missed rate, and alert response time for a realistic traffic scenario. Based on experience using live data, the concept of a valid alert is introduced to deal with an alert that is not associated with an actual conflict nor considered to be a nuisance alert (e.g., an alert issued prior to an aircraft maneuver that avoids loss of separation). To classify alerts as nuisance or valid, and to check timeliness of alerts associated with a conflict, the approach relies on the comparison of the alerts declared by the system with those that would be expected from “truth data” projected forward in time (linear predictor) — truth data defined as the actual aircraft paths. Detailed alert classification rules addressing issues encountered in performance testing with realistic scenario data are described. Approaches to obtaining a representation of “truth data” are referenced. The method of using a Test Predictor operating on truth data in association with alert classification rules was used in performance evaluation of, and problem identification related to, the tactical alert function in the En Route Automation Modernization (ERAM) system and for studies of the Common Automated Radar Terminal System (Common ARTS). Aspects of the performance measurement approach described herein may be applicable to the development of accuracy requirements of future systems.