Online least-squares training for the underdetermined case

We describe an online method of training neural networks, which is based on solving the linearized least-squares problem using the pseudo-inverse for the underdetermined case. This underdetermined linearized least squares (ULLS) method requires significantly less computation and memory for implementation than standard higher-order methods such as the Gauss-Newton method or extended Kalman filter. This decrease is possible because the method allows training to proceed with a smaller number of samples than parameters. Simulation results which compare the performance of the ULLS algorithm to the recursive linearized least squares algorithm (RLLS) and the gradient descent algorithm are presented. Results showing the impact on computational complexity and squared-error performance of the ULLS method, when the number of terms in the Jacobian matrix is varied, are also presented.