Convergence analysis of gradient flow for overparameterized LQR formulations

Abstract

This paper analyzes the intersection between results from gradient methods for the model-free linear quadratic regulator (LQR) problem, and linear feedforward neural networks (LFFNNs). More specifically, it looks into the case where one wants to find an LFFNN feedback that minimizes an LQR cost. It starts by deriving a key conservation law of the system, which is then leveraged to generalize existing results on boundedness and global convergence of solutions, and invariance of the set of stabilizing LFFNNs under the training dynamics (gradient flow). For the single hidden layer LFFNN, the paper proves that the solution converges to the optimal feedback control law for all but a set of Lebesgue measure zero of the initializations. These results are followed by an analysis of a simple version of the problem – the “vector case” – proving the theoretical properties of accelerated convergence and a type of input-to-state stability (ISS) result for this simpler example. Finally, the paper presents numerical evidence of faster convergence of the gradient flow of general LFFNNs when compared to non-overparameterized formulations, showing that the acceleration of the solution is observable even when the gradient is not explicitly computed, but estimated from evaluations of the cost function.