Minimax rates of convergence for high-dimensional regression under ℓq-ball sparsity

Abstract

Consider the standard linear regression model y = Xß∗ + w, where y ∊ Rⁿ is an observation vector, X ∊ R^n×d is a measurement matrix, ß∗ ∊ R^d is the unknown regression vector, and w ~ N (0, σ²Ι) is additive Gaussian noise. This paper determines sharp minimax rates of convergence for estimation of ß∗ in l2 norm, assuming that β∗ belongs to a weak lb-ball Bq(ñq) for some q ∊ [0,1]. We show that under suitable regularity conditions on the design matrix X, the minimax error in squared l2-norm scales as Rq(log d ÷ n)^{1 −q÷2}. In addition, we provide lower bounds on rates of convergence for general lp norm (for all p ∊ [l,+∞], p ≠ q). Our proofs of the lower bounds are information-theoretic in nature, based on Fano's inequality and results on the metric entropy of the balls Bq(Rq). Matching upper bounds are derived by direct analysis of the solution to an optimization algorithm over Bq(Rq). We prove that the conditions on X required by optimal algorithms are satisfied with high probability by broad classes of non-i.i.d. Gaussian random matrices, for which RIP or other sparse eigenvalue conditions are violated. For q = 0, t1-based methods (Lasso and Dantzig selector) achieve the minimax optimal rates in t2 error, but require stronger regularity conditions on the design than the non-convex optimization algorithm used to determine the minimax upper bounds.