Analysis and comparison of two-level KFAC methods for training deep neural networks

IF 1.4 3区数学 Q3 COMPUTER SCIENCE, SOFTWARE ENGINEERING

Optimization Methods & Software Pub Date : 2024-07-24 DOI:10.1080/10556788.2024.2380684

Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Valérie Garès, Mounir Haddou, Quang Huy Tran

引用次数: 0

Abstract

As a second-order method, the Natural Gradient Descent (NGD) has the ability to accelerate training of neural networks. However, due to the prohibitive computational and memory costs of computing a...

查看原文本刊更多论文

用于训练深度神经网络的两级 KFAC 方法的分析与比较

作为一种二阶方法，自然梯度下降法（NGD）能够加速神经网络的训练。然而，由于计算自然梯度下降（NGD）所需的计算成本和内存成本过高，NGD 无法满足神经网络的训练需求。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Optimization Methods & Software 工程技术-计算机：软件工程

CiteScore

4.50

自引率

0.00%

发文量

审稿时长

7 months

期刊介绍： Optimization Methods and Software publishes refereed papers on the latest developments in the theory and realization of optimization methods, with particular emphasis on the interface between software development and algorithm design. Topics include: Theory, implementation and performance evaluation of algorithms and computer codes for linear, nonlinear, discrete, stochastic optimization and optimal control. This includes in particular conic, semi-definite, mixed integer, network, non-smooth, multi-objective and global optimization by deterministic or nondeterministic algorithms. Algorithms and software for complementarity, variational inequalities and equilibrium problems, and also for solving inverse problems, systems of nonlinear equations and the numerical study of parameter dependent operators. Various aspects of efficient and user-friendly implementations: e.g. automatic differentiation, massively parallel optimization, distributed computing, on-line algorithms, error sensitivity and validity analysis, problem scaling, stopping criteria and symbolic numeric interfaces. Theoretical studies with clear potential for applications and successful applications of specially adapted optimization methods and software to fields like engineering, machine learning, data mining, economics, finance, biology, or medicine. These submissions should not consist solely of the straightforward use of standard optimization techniques.