Very many variables and limited numbers of observations; The p>>n problem in current statistical applications

Abstract

Summary form only given. Nonlinearity and chaos are ubiquitous and fascinating. Chaotic systems, in particular, are exquisitely sensitive to small perturbations, but their behavior has a fixed and highly characteristic pattern. Understanding this somewhat counterintuitive combination of effects is important to one's ability to model the physical world. I will begin this talk by reviewing of some of the basic ideas of the field of nonlinear dynamics and describe how those ideas can be leveraged to analyze time-series data. Most of these nonlinear time-series analysis techniques were developed for low-dimensional systems, however, and many of them require perfect models — situations that are rare in the geosciences. For practitioners in these fields, then, it is important to understand how and when to use nonlinear time-series analysis, how to interpret the results, and how to recognize when and why these methods fail. I will demonstrate all of this in the context of a specific problem: understanding and predicting processor and memory loads in