From the simplex to the sphere: faster constrained optimization using the Hadamard parametrization

The standard simplex in $\mathbb{R}^{n}$, also known as the probability simplex, is the set of nonnegative vectors whose entries sum up to 1. It frequently appears as a constraint in optimization problems that arise in machine learning, statistics, data science, operations research and beyond. We convert the standard simplex to the unit sphere and thus transform the corresponding constrained optimization problem into an optimization problem on a simple, smooth manifold. We show that Karush-Kuhn-Tucker points and strict-saddle points of the minimization problem on the standard simplex all correspond to those of the transformed problem, and vice versa. So, solving one problem is equivalent to solving the other problem. Then, we propose several simple, efficient and projection-free algorithms using the manifold structure. The equivalence and the proposed algorithm can be extended to optimization problems with unit simplex, weighted probability simplex or $\ell _{1}$-norm sphere constraints. Numerical experiments between the new algorithms and existing ones show the advantages of the new approach. Open source code is available at https://github.com/DanielMckenzie/HadRGD.