A saddlepoint approximation for the smoothed periodogram

Abstract

Poor variance properties of the periodogram often limit its practical applicability to a wide range of modern spectral estimation and detection applications. The smoothed periodogram, a refined periodogram-based method, is one such nonparametric approach to reducing variance. Neighboring spectral samples are averaged across a spectral window, as opposed to the more common temporal or lag window. Tapered spectral windows and other modifications needed to address the time-bandwidth product and resolution-variance trade-offs complicate the statistical analysis, making it difficult to quantify statistical performance. In addition, approximate distributions for the smoothed periodogram require a priori normalization along with simplifying assumptions to yield computationally tractable results. Here, under mild asymptotic conditions, the distribution derived prior to normalization is shown to be computationally intractable in most cases. First-order statistical approximations are computationally stable but result in sizeable inaccuracies, particularly in the tails. We use a saddlepoint approximation, a second-order asymptotic method, that allows for accurate statistical characterization but is also numerically stable. Monte Carlo simulations are used to validate the results and to illustrate the robustness of the approach. Finally, its utility is demonstrated on a real-world dataset relevant to the side-channel and hardware cybersecurity communities.