Improving Automatic Target Recognition (ATR) Performance with Electro Optics (EO) and Infrared (IR) Sensor Fusion

In previous years, we developed Automatic Target Recognition (ATR) algorithms by combining layers of the open-source You Only Look Once (YOLOv2) detection model with customized Convolutional Neural Network (CNN) feature extraction layers to recognize targets in Infrared (IR) images. Our work showed that ATR for IR performed significantly better during night-time than during daytime. In this study, we demonstrate that fusing EO and IR images using pixel-based and decision-based sensor fusion can improve daytime ATR performance significantly. Traditional Automatic Target Detection (ATD) metrics do not account for misclassification while traditional target classification metrics do not count missed detections. We have developed a novel approach for evaluating ATR performance that bridges traditional target detection and target classification metrics based on an extended confusion matrix (ECM) which allows us to accurately characterize Probability of Detection (Pd), Probability of False Alarm (Pfa), and the tradeoffs between the two for ATR applications. After running the ATR detector at multiple Confidence Score thresholds, we can obtain and describe the detection performance at different Pfa levels using the Receiver Operating Characteristic (ROC) curves to show the comprehensive relationship between Pd vs. Pfa. Combining EO and IR fusion approaches with the ECM, we demonstrate improvements in Pd of 11% with pixel-based fusion and 13-17% with decision-based fusion, respectively.