Efficient Optimization of Logistic Regression by Direct Use of Conjugate Gradient

Abstract

In classification problems, logistic regression (LR) is used to estimate posterior probabilities. The objective function of LR is usually minimized by Newton-Raphson method such as using iterative reweighted least squares (IRLS). There, the inverse Hessian matrix must be calculated in each iteration step. Thus, a computational cost in the optimization of LR significantly increases as input data becomes large. To reduce the computational cost, we propose a novel optimization method of LR by directly using the non-linear conjugate gradient (CG) method. The proposed method iteratively minimizes the objective function of LR without calculation of the Hessian matrix. Furthermore, to reduce the number of iteration efficiently, the step size in the non-linear CG iteration is optimized avoiding ad hock line search, and initial values are set by ordinary linear regression analysis. In the experimental results, our method performs about 200 times faster than the other methods for a large scale dataset.