Variable Projection algorithms: Theoretical insights and a novel approach for problems with large residual

IF 4.8 2区计算机科学 Q1 AUTOMATION & CONTROL SYSTEMS

Automatica Pub Date : 2025-04-14 DOI:10.1016/j.automatica.2025.112300

Guangyong Chen , Peng Xue , Min Gan , Jing Chen , Wenzhong Guo , C.L. Philip Chen

{"title":"Variable Projection algorithms: Theoretical insights and a novel approach for problems with large residual","authors":"Guangyong Chen , Peng Xue , Min Gan , Jing Chen , Wenzhong Guo , C.L. Philip Chen","doi":"10.1016/j.automatica.2025.112300","DOIUrl":null,"url":null,"abstract":"<div><div>This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification, image processing, and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman’s VP algorithm can achieve a similar convergence rate as the Golub & Pereyra’s form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, inspired by these theoretical insights, we design a refined VP algorithm, termed VPLR, to address separable nonlinear optimization problems with large residual. This algorithm enhances convergence performance by addressing the coupling relationship between parameters in separable models and continually refining the approximated Hessian matrix to counteract the influence of large residual. The effectiveness of this refined algorithm is corroborated through numerical experiments.</div></div>","PeriodicalId":55413,"journal":{"name":"Automatica","volume":"177 ","pages":"Article 112300"},"PeriodicalIF":4.8000,"publicationDate":"2025-04-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Automatica","FirstCategoryId":"94","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S000510982500192X","RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"AUTOMATION & CONTROL SYSTEMS","Score":null,"Total":0}

引用次数: 0

Abstract

This paper delves into an in-depth exploration of the Variable Projection (VP) algorithm, a powerful tool for solving separable nonlinear optimization problems across multiple domains, including system identification, image processing, and machine learning. We first establish a theoretical framework to examine the effect of the approximate treatment of the coupling relationship among parameters on the local convergence of the VP algorithm and theoretically prove that the Kaufman’s VP algorithm can achieve a similar convergence rate as the Golub & Pereyra’s form. These studies fill the gap in the existing convergence theory analysis, and provide a solid foundation for understanding the mechanism of VP algorithm and broadening its application horizons. Furthermore, inspired by these theoretical insights, we design a refined VP algorithm, termed VPLR, to address separable nonlinear optimization problems with large residual. This algorithm enhances convergence performance by addressing the coupling relationship between parameters in separable models and continually refining the approximated Hessian matrix to counteract the influence of large residual. The effectiveness of this refined algorithm is corroborated through numerical experiments.

查看原文本刊更多论文

变量投影算法：理论见解和处理大残差问题的新方法

本文深入探讨了变量投影（VP）算法，这是一个强大的工具，用于解决跨多个领域的可分离非线性优化问题，包括系统识别，图像处理和机器学习。我们首先建立了理论框架，考察了参数间耦合关系的近似处理对VP算法局部收敛性的影响，并从理论上证明了Kaufman的VP算法可以达到与Golub &；Pereyra的形式。这些研究填补了现有收敛理论分析的空白，为理解VP算法的机理和拓宽其应用领域提供了坚实的基础。此外，受这些理论见解的启发，我们设计了一种改进的VP算法，称为VPLR，以解决具有大残差的可分离非线性优化问题。该算法通过解决可分离模型中参数之间的耦合关系，并不断改进逼近的Hessian矩阵来抵消大残差的影响，从而提高了收敛性能。通过数值实验验证了该改进算法的有效性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Automatica 工程技术-工程：电子与电气

CiteScore

10.70

自引率

7.80%

发文量

617

审稿时长

5 months

期刊介绍： Automatica is a leading archival publication in the field of systems and control. The field encompasses today a broad set of areas and topics, and is thriving not only within itself but also in terms of its impact on other fields, such as communications, computers, biology, energy and economics. Since its inception in 1963, Automatica has kept abreast with the evolution of the field over the years, and has emerged as a leading publication driving the trends in the field. After being founded in 1963, Automatica became a journal of the International Federation of Automatic Control (IFAC) in 1969. It features a characteristic blend of theoretical and applied papers of archival, lasting value, reporting cutting edge research results by authors across the globe. It features articles in distinct categories, including regular, brief and survey papers, technical communiqués, correspondence items, as well as reviews on published books of interest to the readership. It occasionally publishes special issues on emerging new topics or established mature topics of interest to a broad audience. Automatica solicits original high-quality contributions in all the categories listed above, and in all areas of systems and control interpreted in a broad sense and evolving constantly. They may be submitted directly to a subject editor or to the Editor-in-Chief if not sure about the subject area. Editorial procedures in place assure careful, fair, and prompt handling of all submitted articles. Accepted papers appear in the journal in the shortest time feasible given production time constraints.