SIAM Journal on Optimization最新文献_第3页

Provably Faster Gradient Descent via Long Steps 通过长步骤实现更快的梯度下降

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-18 DOI: 10.1137/23m1588408

Benjamin Grimmer

引用次数: 0

Fast Gradient Algorithm with Dry-like Friction and Nonmonotone Line Search for Nonconvex Optimization Problems 针对非凸优化问题的干摩擦和非单调线性搜索快速梯度算法

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-17 DOI: 10.1137/22m1532354

Lien T. Nguyen, Andrew Eberhard, Xinghuo Yu, Chaojie Li

{"title":"Fast Gradient Algorithm with Dry-like Friction and Nonmonotone Line Search for Nonconvex Optimization Problems","authors":"Lien T. Nguyen, Andrew Eberhard, Xinghuo Yu, Chaojie Li","doi":"10.1137/22m1532354","DOIUrl":"https://doi.org/10.1137/22m1532354","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2557-2587, September 2024. Abstract. In this paper, we propose a fast gradient algorithm for the problem of minimizing a differentiable (possibly nonconvex) function in Hilbert spaces. We first extend the dry friction property for convex functions to what we call the dry-like friction property in a nonconvex setting, and then employ a line search technique to adaptively update parameters at each iteration. Depending on the choice of parameters, the proposed algorithm exhibits subsequential convergence to a critical point or full sequential convergence to an “approximate” critical point of the objective function. We also establish the full sequential convergence to a critical point under the Kurdyka–Łojasiewicz (KL) property of a merit function. Thanks to the parameters’ flexibility, our algorithm can reduce to a number of existing inertial gradient algorithms with Hessian damping and dry friction. By exploiting variational properties of the Moreau envelope, the proposed algorithm is adapted to address weakly convex nonsmooth optimization problems. In particular, we extend the result on KL exponent for the Moreau envelope of a convex KL function to a broad class of KL functions that are not necessarily convex nor continuous. Simulation results illustrate the efficiency of our algorithm and demonstrate the potential advantages of combining dry-like friction with extrapolation and line search techniques.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":"159 1","pages":""},"PeriodicalIF":3.1,"publicationDate":"2024-07-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141719096","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Finitely Convergent Circumcenter Method for the Convex Feasibility Problem 凸可行性问题的有限收敛圆心法

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-15 DOI: 10.1137/23m1595412

Roger Behling, Yunier Bello-Cruz, Alfredo N. Iusem, Di Liu, Luiz-Rafael Santos

引用次数: 0

Using Taylor-Approximated Gradients to Improve the Frank–Wolfe Method for Empirical Risk Minimization 使用泰勒近似梯度改进经验风险最小化的弗兰克-沃尔夫方法

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-11 DOI: 10.1137/22m1519286

Zikai Xiong, Robert M. Freund

{"title":"Using Taylor-Approximated Gradients to Improve the Frank–Wolfe Method for Empirical Risk Minimization","authors":"Zikai Xiong, Robert M. Freund","doi":"10.1137/22m1519286","DOIUrl":"https://doi.org/10.1137/22m1519286","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2503-2534, September 2024. Abstract. The Frank–Wolfe method has become increasingly useful in statistical and machine learning applications due to the structure-inducing properties of the iterates and especially in settings where linear minimization over the feasible set is more computationally efficient than projection. In the setting of empirical risk minimization—one of the fundamental optimization problems in statistical and machine learning—the computational effectiveness of Frank–Wolfe methods typically grows linearly in the number of data observations [math]. This is in stark contrast to the case for typical stochastic projection methods. In order to reduce this dependence on [math], we look to second-order smoothness of typical smooth loss functions (least squares loss and logistic loss, for example), and we propose amending the Frank–Wolfe method with Taylor series–approximated gradients, including variants for both deterministic and stochastic settings. Compared with current state-of-the-art methods in the regime where the optimality tolerance [math] is sufficiently small, our methods are able to simultaneously reduce the dependence on large [math] while obtaining optimal convergence rates of Frank–Wolfe methods in both convex and nonconvex settings. We also propose a novel adaptive step-size approach for which we have computational guarantees. Finally, we present computational experiments which show that our methods exhibit very significant speedups over existing methods on real-world datasets for both convex and nonconvex binary classification problems.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":"36 1","pages":""},"PeriodicalIF":3.1,"publicationDate":"2024-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141613419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Complexity of Finite-Sum Optimization with Nonsmooth Composite Functions and Non-Lipschitz Regularization 具有非光滑复合函数和非 Lipschitz 正则化的有限和优化的复杂性

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-10 DOI: 10.1137/23m1546701

Xiao Wang, Xiaojun Chen

{"title":"Complexity of Finite-Sum Optimization with Nonsmooth Composite Functions and Non-Lipschitz Regularization","authors":"Xiao Wang, Xiaojun Chen","doi":"10.1137/23m1546701","DOIUrl":"https://doi.org/10.1137/23m1546701","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2472-2502, September 2024. Abstract. In this paper, we present complexity analysis of proximal inexact gradient methods for finite-sum optimization with a nonconvex nonsmooth composite function and non-Lipschitz regularization. By getting access to a convex approximation to the Lipschitz function and a Lipschitz continuous approximation to the non-Lipschitz regularizer, we construct a proximal subproblem at each iteration without using exact function values and gradients. With certain accuracy control on inexact gradients and subproblem solutions, we show that the oracle complexity in terms of total number of inexact gradient evaluations is in order [math] to find an [math]-approximate first-order stationary point, ensuring that within a [math]-ball centered at this point the maximum reduction of an approximation model does not exceed [math]. This shows that we can have the same worst-case evaluation complexity order as in [C. Cartis, N. I. M. Gould, and P. L. Toint, SIAM J. Optim., 21 (2011), pp. 1721–1739, X. Chen, Ph. L. Toint, and H. Wang, SIAM J. Optim., 29 (2019), pp. 874–903], even if we introduce the non-Lipschitz singularity and the nonconvex nonsmooth composite function in the objective function. Moreover, we establish that the oracle complexity regarding the total number of stochastic oracles is in order [math] with high probability for stochastic proximal inexact gradient methods. We further extend the algorithm to adjust to solving stochastic problems with expectation form and derive the associated oracle complexity in order [math] with high probability.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":"152 1","pages":""},"PeriodicalIF":3.1,"publicationDate":"2024-07-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141585848","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

The Rate of Convergence of Bregman Proximal Methods: Local Geometry Versus Regularity Versus Sharpness 布雷格曼近端方法的收敛速度：局部几何VS正规性VS锐利性

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-09 DOI: 10.1137/23m1580218

Waïss Azizian, Franck Iutzeler, Jérôme Malick, Panayotis Mertikopoulos

{"title":"The Rate of Convergence of Bregman Proximal Methods: Local Geometry Versus Regularity Versus Sharpness","authors":"Waïss Azizian, Franck Iutzeler, Jérôme Malick, Panayotis Mertikopoulos","doi":"10.1137/23m1580218","DOIUrl":"https://doi.org/10.1137/23m1580218","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2440-2471, September 2024. Abstract. We examine the last-iterate convergence rate of Bregman proximal methods—from mirror descent to mirror-prox and its optimistic variants—as a function of the local geometry induced by the prox-mapping defining the method. For generality, we focus on local solutions of constrained, nonmonotone variational inequalities, and we show that the convergence rate of a given method depends sharply on its associated Legendre exponent, a notion that measures the growth rate of the underlying Bregman function (Euclidean, entropic, or other) near a solution. In particular, we show that boundary solutions exhibit a stark separation of regimes between methods with a zero and nonzero Legendre exponent: The former converge at a linear rate, while the latter converge, in general, sublinearly. This dichotomy becomes even more pronounced in linearly constrained problems where methods with entropic regularization achieve a linear convergence rate along sharp directions, compared to convergence in a finite number of steps under Euclidean regularization.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":"67 1","pages":""},"PeriodicalIF":3.1,"publicationDate":"2024-07-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141575207","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Subgradient Regularized Multivariate Convex Regression at Scale 子梯度正则化多变量尺度凸回归

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-02 DOI: 10.1137/21m1413134

Wenyu Chen, Rahul Mazumder

{"title":"Subgradient Regularized Multivariate Convex Regression at Scale","authors":"Wenyu Chen, Rahul Mazumder","doi":"10.1137/21m1413134","DOIUrl":"https://doi.org/10.1137/21m1413134","url":null,"abstract":"SIAM Journal on Optimization, Volume 34, Issue 3, Page 2350-2377, September 2024. Abstract. We present new large-scale algorithms for fitting a subgradient regularized multivariate convex regression function to [math] samples in [math] dimensions—a key problem in shape constrained nonparametric regression with applications in statistics, engineering, and the applied sciences. The infinite-dimensional learning task can be expressed via a convex quadratic program (QP) with [math] decision variables and [math] constraints. While instances with [math] in the lower thousands can be addressed with current algorithms within reasonable runtimes, solving larger problems (e.g., [math] or [math]) is computationally challenging. To this end, we present an active set type algorithm on the dual QP. For computational scalability, we allow for approximate optimization of the reduced subproblems and propose randomized augmentation rules for expanding the active set. We derive novel computational guarantees for our algorithms. We demonstrate that our framework can approximately solve instances of the subgradient regularized convex regression problem with [math] and [math] within minutes and shows strong computational performance compared to earlier approaches.","PeriodicalId":49529,"journal":{"name":"SIAM Journal on Optimization","volume":"237 1","pages":""},"PeriodicalIF":3.1,"publicationDate":"2024-07-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141519736","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Descent Algorithm for the Optimal Control of ReLU Neural Network Informed PDEs Based on Approximate Directional Derivatives 基于近似方向衍生物的 ReLU 神经网络 PDE 最佳控制后裔算法

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-02 DOI: 10.1137/22m1534420

Guozhi Dong, Michael Hintermüller, Kostas Papafitsoros

引用次数: 0

Fast Convergence of Inertial Multiobjective Gradient-Like Systems with Asymptotic Vanishing Damping 具有渐近消失阻尼的惯性多目标梯度样系统的快速收敛性

IF 3.1 1区数学

SIAM Journal on Optimization Pub Date : 2024-07-02 DOI: 10.1137/23m1588512

Konstantin Sonntag, Sebastian Peitz

引用次数: 0

Scalable Frank–Wolfe on Generalized Self-Concordant Functions via Simple Steps 通过简单步骤对广义自洽函数进行可扩展的弗兰克-沃尔夫计算