Revisiting the central limit theorems for the SGD-type methods

IF 16.4 1区化学 Q1 CHEMISTRY, MULTIDISCIPLINARY

Accounts of Chemical Research Pub Date : 2024-07-15 DOI:10.4310/cms.2024.v22.n5.a10

Tiejun Li, Tiannan Xiao, Guoguo Yang

引用次数: 0

Abstract

We revisited the central limit theorem (CLT) for stochastic gradient descent (SGD) type methods, including the vanilla SGD, momentum SGD and Nesterov accelerated SGD methods with constant or vanishing damping parameters. By taking advantage of Lyapunov function technique and $L^p$ bound estimates, we established the CLT under more general conditions on learning rates for broader classes of SGD methods as compared to previous results. The CLT for the time average was also investigated, and we found that it held in the linear case, while it was not generally true in nonlinear situation. Numerical tests were also carried out to verify our theoretical analysis.

查看原文本刊更多论文

重温 SGD 类方法的中心极限定理

我们重新审视了随机梯度下降（SGD）类型方法的中心极限定理（CLT），包括具有恒定或消失阻尼参数的虚无 SGD、动量 SGD 和内斯特洛夫加速 SGD 方法。通过利用 Lyapunov 函数技术和 $L^p$ 边界估计，与之前的结果相比，我们为更广泛类别的 SGD 方法建立了学习率条件下的 CLT。我们还研究了时间平均的 CLT，发现它在线性情况下成立，而在非线性情况下一般不成立。我们还进行了数值测试，以验证我们的理论分析。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Accounts of Chemical Research 化学-化学综合

CiteScore

31.40

自引率

1.10%

发文量

312

审稿时长

2 months

期刊介绍： Accounts of Chemical Research presents short, concise and critical articles offering easy-to-read overviews of basic research and applications in all areas of chemistry and biochemistry. These short reviews focus on research from the author’s own laboratory and are designed to teach the reader about a research project. In addition, Accounts of Chemical Research publishes commentaries that give an informed opinion on a current research problem. Special Issues online are devoted to a single topic of unusual activity and significance. Accounts of Chemical Research replaces the traditional article abstract with an article "Conspectus." These entries synopsize the research affording the reader a closer look at the content and significance of an article. Through this provision of a more detailed description of the article contents, the Conspectus enhances the article's discoverability by search engines and the exposure for the research.