Evaluation of ATR algorithms employing motion imagery

Most Automated Target Recognition (ATR) algorithms developed for Intelligence, Surveillance, and Reconnaissance (ISR) missions operate on a single frame of still imagery to detect, recognize, and geolocate targets of interest. The introduction of digital motion imagery for ISR applications raises the need for automated tools to assist the image analyst (IA). Furthermore, the temporal information and frequent revisit available from motion imagery facilitates the extraction of information not previously available to the IA. Consequently, the evaluation methods needed for assessing the performance of ATR processing of motion imagery extend beyond the framework employed in traditional ATR evaluations. This paper presents the issues associated with evaluations of ATR algorithms for motion imagery and develops approaches for addressing these issues. The major issues fall into three broad categories: Characterization of the testing problem: The concepts of standard operating conditions and extended operating conditions, which are used to distinguish "easy" ATR problems from "hard" ones, require some modifications for motion imagery. For example, targets in the clear could prove challenging if target density is high or vehicle tracks cross frequently. Developing image truth and scoring rules: The introduction of the temporal dimension raises some ambiguities about what constitutes successful target detection-is it necessary to detect and track a vehicle through a full video clip or is detection on a single frame sufficient? Performance metrics: New performance metrics, that go beyond simple detection, identification, and false alarm rates, are needed to characterize performance in the context of motion imagery. We propose an approach to quantify battlefield awareness, based on simple measures of performance.