Lifting the curse of dimensionality: a random matrix-theoretic approach

Abstract

The ubiquity of inexpensive sensors implies that we can measure vector-valued data of ever increasing dimension. But the number of independent measurements of the data vector is limited so the sample covariance matrix is usually singular. The traditional remedy for singularity is diagonal loading - the addition of a small identity matrix to make the covariance estimate invertible. An alternative to diagonal loading is to reduce the dimension of the data vectors to be smaller than the number of independent observations through an ensemble of isotropically random (Haar measure) unitary matrices. For every member of the unitary ensemble, the shortened data vectors yield a statistically meaningful, invertible covariance estimate from which we can compute an estimate for the ultimate desired quantity. The final step is to take the expectation of this estimate with respect to the unitary ensemble. For a class of applications that includes adaptive spectral estimation, the design of a linear estimator, and supervised learning the random matrix approach results in an estimate for the inverse covariance matrix which preserves the eigenvectors of the sample covariance matrix, but alters the eigenvalues in a nontrivial manner. A closed-form expression for the expectation over the unitary ensemble eludes us, but we have obtained a tractable asymptotic expression. Preliminary numerical results indicate considerable promise for this approach.