Global Optimality of the EM Algorithm for Mixtures of Two-Component Linear Regressions

Abstract

Recent results established that EM enjoys global convergence for Gaussian Mixture Models. For Mixed Linear Regression, however, only local convergence results have been established, and those only for the high signal-to-noise ratio (SNR) regime. In this work, we completely characterize the global optimality of EM: we show that starting from any randomly initialized point, the EM algorithm converges to the true parameter ${\beta }^{*}$ at the minimax statistical rates under all SNR regimes. Toward this goal, we first show the global convergence of the EM algorithm at the population level. Then we provide a complete characterization of statistical and computational behaviors of EM under all SNR regimes with finite samples. In particular: (i) When the SNR is sufficiently large, the EM updates converge to the true parameter $ {\beta }^{*}$ at the standard parametric convergence rate $O((d/n)^{1/2})$ after $O(\log (n/d))$ iterations. (ii) In the regime where the SNR is above $O((d/n)^{1/4})$ and below some constant, the EM iterates converge to a $O({\mathrm { SNR}}^{-1} (d/n)^{1/2})$ neighborhood of the true parameter, when the number of iterations is of the order $O({\mathrm { SNR}}^{-2} \log (n/d))$ . (iii) In the low SNR regime where the SNR is below $O((d/n)^{1/4})$ , we show that EM converges to a $O((d/n)^{1/4})$ neighborhood of the true parameters, after $O((n/d)^{1/2})$ iterations. By providing tight convergence guarantees of the EM algorithm in middle-to-low SNR regimes, we reveal that in low SNR, EM changes rate, matching the $n^{-1/4}$ rate of the MLE, a behavior that previous work had been unable to show.