Stochastic Gradient Descent Combines Second-Order Information for Training Neural Network

Deep learning is received special attention in the last decade following the increasing popularity of artificial intelligence. A successful deep learning application highly depends on an effective training neural network method. Currently, the first-order methods, e.g. stochastic gradient descent method may be the most widely-used method due to its simplicity and generally good performance. However, the first methods possess varied weakness, like lower convergence rate and easily stalking around saddle points for the nonconvex neural network problem. The second-order method, on the other hand, can address these issues by utilizing second derivative information, but the high computational cost of computing second-derivative information limits its usage. Based on these motivations, we design a new training schema that combine the advantages of first and second methods, meanwhile eliminate their disadvantages. To demonstrate its effectiveness, we test the new method on dataset, cifar-10. The results show the new approach performs as our desired.