A Review of Well-Known Robust Line Search and Trust Region Numerical Optimization Algorithms for Solving Nonlinear Least-Squares Problems

The conditional, unconditional, or the exact maximum likelihood estimation and the least-squares estimation involve minimizing either the conditional or the unconditional residual sum of squares. The maximum likelihood estimation (MLE) approach and the nonlinear least squares (NLS) procedure involve an iterative search technique for obtaining global rather than local optimal estimates. Several authors have presented brief overviews of algorithms for solving NLS problems. Snezana S. Djordjevic (2019) presented a review of some unconstrained optimization methods based on the line search techniques. Mahaboob et al. (2017) proposed a different approach to estimate nonlinear regression models using numerical methods also based on the line search techniques. Mohammad, Waziri, and Santos (2019) have briefly reviewed methods for solving NLS problems, paying special attention to the structured quasi-Newton methods which are the family of the search line techniques. Ya-Xiang Yuan (2011) reviewed some recent results on numerical methods for nonlinear equations and NLS problems based on online searches and trust regions techniques, particularly on Levenberg-Marquardt type methods, quasi-Newton type methods, and trust regions algorithms. The purpose of this paper is to review some online searches and trust region's more well-known robust numerical optimization algorithms and the most used in practice for the estimation of time series models and other nonlinear regression models. The line searches algorithms considered are: Gradient algorithm, Steepest Descent (SD) algorithm, Newton-Raphson (NR) algorithm, Murray’s algorithm, Quasi-Newton (QN) algorithm, Gauss-Newton (GN) algorithm, Fletcher and Powell algorithm (FP), Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. While the only trust-region algorithm considered is the Levenberg-Marquardt (LM) algorithm. We also give some main advantages and disadvantages of these different algorithms.