Second-Order Asymptotically Optimal Outlying Sequence Detection with Reject Option

Motivated by practical machine learning applications, we revisit the outlying sequence detection problem (Li et al., TIT 2014) and derive fundamental limits of optimal detection when the reject option is allowed for outlying sequences. In the considered outlying sequence detection (OSD) problem, one is given multiple observed sequences, where all sequences are generated i.i.d. from a nominal distribution with at most one exception. The task is to discern the outlying sequence that is generated according to an anomalous distribution. In OSD, the nominal and anomalous distributions are unknown. In this paper, we consider the case where there is a reject option for the OSD, i.e., we reject the samples as insufficient for making a reliable decision (cf. Bartlett et al., JMLR 2008). We study the tradeoff among the probabilities of misclassification error, false alarm and false reject for tests that satisfy weak conditions on the rate of decrease of these error probabilities as a function of sequence length. We propose a second-order asymptotically optimal test that provides a finite sample approximation to the error probabilities.