Performance of Sequential and Quickest Detection Based on Machine Learning

Abstract

With the rapid proliferation of Machine Learning (ML) and Deep Learning (DL) based decision systems, properly characterizing their often unpredictable performance is a key challenge. In this work we introduce data-driven analogues to the Sequential Probability Ratio Test (SPRT), and Page Test, denoted the Sequential Data-Driven Decision Function (S-D3F) and Quickest Data-Driven Decision Function (Q-D3F), respectively. It is shown that the S-D3F and Q-D3F inherit many key properties from the SPRT and Page test which allow for a reliable extension of their performance prediction strategies to these data-driven approaches via the process of statistical characterization. As a result, one is able to perform sequential analysis and quickest detection using a ML driven detector while attaining near-optimal performance and reliable performance modeling; that is, the performance can be predicted from the training data itself. The reliability of these predictions is demonstrated via extensive numerical experimentation, including a challenging low-SNR passive acoustic detection problem via an ambient acoustic dataset injected with a realistic synthetic signal.