Finding a Needle in a Haystack

Abstract

Summary form only given. We study a simplified version of the problem of target detectability in the presence of clutter. The target (the needle) is a sample of size N from a discrete distribution p. The clutter (the haystack) is made up of M independent samples of size JV from a distribution q (which is different from p, but with the same support). Two cases can be easily shown: (i) If M is fixed and JV goes to infinity, the target can be detected with probability that approaches 1. (ii) If TV is fixed and M goes to infinity, then, with probability approaching 1, the target cannot be detected. For the case where both JV, M go to infinity, we show that the asymptotic behavior of the optimal detector (if p, q are known) and of a plug-in detector (which estimates p, q on the fly) is determined by the asymptotic behavior of the quantity Mexp(-ND(p\\q)) : if it goes to zero (resp. infinity), then, with high probability, the target can (resp. cannot) be detected.