Nearly-Tight Bounds for Private Stochastic Convex Optimization and Sparse Learning Under Heavy Tails

Abstract

We study differentially private stochastic convex optimization (DP-SCO) under heavy-tailed data, moving beyond standard assumptions like Lipschitz losses or gradients with bounded second moments. We focus on the setting where each gradient coordinate has only a bounded $(1+v)$-th moment for some $v\in(0,1]$, allowing much heavier tails. We first propose an optimal private and robust mean estimator for the one-dimensional case, with matching upper and lower bounds. We then extend this to $d$ dimensions, introducing a new high-dimensional differential privacy (DP) mean estimator with strong accuracy guarantees. A key technical result is a new concentration inequality for distributions with bounded $(1+v)$-th moments, of independent interest. Leveraging these tools, we design DP-SCO algorithms for convex and strongly convex settings, achieving nearly-tight error bounds. To address high-dimensionality, we study DP-SCO with a sparsity constraint (DP sparse learning), obtaining error rates logarithmic in $d$ and optimal up to a factor of $O(s^{\ast})$, where $s^{\ast}$ is the sparsity level. Finally, we validate our results with numerical experiments.