On Adaptive Stochastic Heavy Ball Momentum for Solving Linear Systems

IF 1.5 2区数学 Q2 MATHEMATICS, APPLIED

SIAM Journal on Matrix Analysis and Applications Pub Date : 2024-07-09 DOI:10.1137/23m1575883

Yun Zeng, Deren Han, Yansheng Su, Jiaxin Xie

{"title":"On Adaptive Stochastic Heavy Ball Momentum for Solving Linear Systems","authors":"Yun Zeng, Deren Han, Yansheng Su, Jiaxin Xie","doi":"10.1137/23m1575883","DOIUrl":null,"url":null,"abstract":"SIAM Journal on Matrix Analysis and Applications, Volume 45, Issue 3, Page 1259-1286, September 2024. <br/> Abstract. The stochastic heavy ball momentum (SHBM) method has gained considerable popularity as a scalable approach for solving large-scale optimization problems. However, one limitation of this method is its reliance on prior knowledge of certain problem parameters, such as singular values of a matrix. In this paper, we propose an adaptive variant of the SHBM method for solving stochastic problems that are reformulated from linear systems using user-defined distributions. Our adaptive SHBM (ASHBM) method utilizes iterative information to update the parameters, addressing an open problem in the literature regarding the adaptive learning of momentum parameters. We prove that our method converges linearly in expectation, with a better convergence bound compared to the basic method. Notably, we demonstrate that the deterministic version of our ASHBM algorithm can be reformulated as a variant of the conjugate gradient (CG) method, inheriting many of its appealing properties, such as finite-time convergence. Consequently, the ASHBM method can be further generalized to develop a brand-new framework of the stochastic CG method for solving linear systems. Our theoretical results are supported by numerical experiments.","PeriodicalId":49538,"journal":{"name":"SIAM Journal on Matrix Analysis and Applications","volume":null,"pages":null},"PeriodicalIF":1.5000,"publicationDate":"2024-07-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"SIAM Journal on Matrix Analysis and Applications","FirstCategoryId":"100","ListUrlMain":"https://doi.org/10.1137/23m1575883","RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MATHEMATICS, APPLIED","Score":null,"Total":0}

引用次数: 0

Abstract

SIAM Journal on Matrix Analysis and Applications, Volume 45, Issue 3, Page 1259-1286, September 2024.
Abstract. The stochastic heavy ball momentum (SHBM) method has gained considerable popularity as a scalable approach for solving large-scale optimization problems. However, one limitation of this method is its reliance on prior knowledge of certain problem parameters, such as singular values of a matrix. In this paper, we propose an adaptive variant of the SHBM method for solving stochastic problems that are reformulated from linear systems using user-defined distributions. Our adaptive SHBM (ASHBM) method utilizes iterative information to update the parameters, addressing an open problem in the literature regarding the adaptive learning of momentum parameters. We prove that our method converges linearly in expectation, with a better convergence bound compared to the basic method. Notably, we demonstrate that the deterministic version of our ASHBM algorithm can be reformulated as a variant of the conjugate gradient (CG) method, inheriting many of its appealing properties, such as finite-time convergence. Consequently, the ASHBM method can be further generalized to develop a brand-new framework of the stochastic CG method for solving linear systems. Our theoretical results are supported by numerical experiments.

查看原文本刊更多论文

论求解线性系统的自适应随机重球动量

SIAM 矩阵分析与应用期刊》，第 45 卷第 3 期，第 1259-1286 页，2024 年 9 月。摘要。随机重球动量（SHBM）方法作为一种解决大规模优化问题的可扩展方法，已经获得了相当高的人气。然而，这种方法的一个局限是它依赖于某些问题参数的先验知识，如矩阵的奇异值。在本文中，我们提出了一种 SHBM 方法的自适应变体，用于解决使用用户定义分布从线性系统重新表述的随机问题。我们的自适应 SHBM（ASHBM）方法利用迭代信息更新参数，解决了文献中关于动量参数自适应学习的一个未决问题。我们证明，我们的方法在期望值上线性收敛，与基本方法相比，收敛约束更好。值得注意的是，我们证明了我们的 ASHBM 算法的确定性版本可以重新表述为共轭梯度（CG）方法的变体，继承了其许多吸引人的特性，如有限时间收敛。因此，ASHBM 方法可以进一步推广，发展出一种全新的用于求解线性系统的随机共轭梯度法框架。我们的理论结果得到了数值实验的支持。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

SIAM Journal on Matrix Analysis and Applications 数学-应用数学

CiteScore

2.90

自引率

6.70%

发文量

审稿时长

6-12 weeks

期刊介绍： The SIAM Journal on Matrix Analysis and Applications contains research articles in matrix analysis and its applications and papers of interest to the numerical linear algebra community. Applications include such areas as signal processing, systems and control theory, statistics, Markov chains, and mathematical biology. Also contains papers that are of a theoretical nature but have a possible impact on applications.