Backpropagation through Back Substitution with a Backslash

IF 1.5 2区数学 Q2 MATHEMATICS, APPLIED

SIAM Journal on Matrix Analysis and Applications Pub Date : 2024-02-05 DOI:10.1137/22m1532871

Alan Edelman, Ekin Akyürek, Yuyang Wang

{"title":"Backpropagation through Back Substitution with a Backslash","authors":"Alan Edelman, Ekin Akyürek, Yuyang Wang","doi":"10.1137/22m1532871","DOIUrl":null,"url":null,"abstract":"SIAM Journal on Matrix Analysis and Applications, Volume 45, Issue 1, Page 429-449, March 2024. <br/> Abstract. We present a linear algebra formulation of backpropagation which allows the calculation of gradients by using a generically written “backslash” or Gaussian elimination on triangular systems of equations. Generally, the matrix elements are operators. This paper has three contributions: (i) it is of intellectual value to replace traditional treatments of automatic differentiation with a (left acting) operator theoretic, graph-based approach; (ii) operators can be readily placed in matrices in software in programming languages such as Julia as an implementation option; (iii) we introduce a novel notation, “transpose dot” operator “[math]” that allows for the reversal of operators. We further demonstrate the elegance of the operators approach in a suitable programming language consisting of generic linear algebra operators such as Julia [Bezanson et al., SIAM Rev., 59 (2017), pp. 65–98], and that it is possible to realize this abstraction in code. Our implementation shows how generic linear algebra can allow operators as elements of matrices. In contrast to “operator overloading,” where backslash would normally have to be rewritten to take advantage of operators, with “generic programming” there is no such need.","PeriodicalId":49538,"journal":{"name":"SIAM Journal on Matrix Analysis and Applications","volume":null,"pages":null},"PeriodicalIF":1.5000,"publicationDate":"2024-02-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"SIAM Journal on Matrix Analysis and Applications","FirstCategoryId":"100","ListUrlMain":"https://doi.org/10.1137/22m1532871","RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"MATHEMATICS, APPLIED","Score":null,"Total":0}

引用次数: 0

Abstract

SIAM Journal on Matrix Analysis and Applications, Volume 45, Issue 1, Page 429-449, March 2024.
Abstract. We present a linear algebra formulation of backpropagation which allows the calculation of gradients by using a generically written “backslash” or Gaussian elimination on triangular systems of equations. Generally, the matrix elements are operators. This paper has three contributions: (i) it is of intellectual value to replace traditional treatments of automatic differentiation with a (left acting) operator theoretic, graph-based approach; (ii) operators can be readily placed in matrices in software in programming languages such as Julia as an implementation option; (iii) we introduce a novel notation, “transpose dot” operator “[math]” that allows for the reversal of operators. We further demonstrate the elegance of the operators approach in a suitable programming language consisting of generic linear algebra operators such as Julia [Bezanson et al., SIAM Rev., 59 (2017), pp. 65–98], and that it is possible to realize this abstraction in code. Our implementation shows how generic linear algebra can allow operators as elements of matrices. In contrast to “operator overloading,” where backslash would normally have to be rewritten to take advantage of operators, with “generic programming” there is no such need.

查看原文本刊更多论文

通过带反斜线的反向置换进行反向传播

SIAM 矩阵分析与应用期刊》，第 45 卷，第 1 期，第 429-449 页，2024 年 3 月。摘要。我们提出了一种反向传播的线性代数公式，它允许在三角方程组中使用通用的 "反斜线 "或高斯消元法计算梯度。一般来说，矩阵元素都是算子。本文有三方面的贡献：(i) 用（左演算）算子理论、基于图的方法取代传统的自动微分处理方法，具有重要的思想价值；(ii) 作为一种实现方案，算子可以很容易地放在软件的矩阵中，如 Julia 等编程语言；(iii) 我们引入了一种新颖的符号，即 "转置点 "算子"[math]"，允许算子反转。我们进一步证明，在由通用线性代数算子组成的合适编程语言（如 Julia）中，算子方法是优雅的[Bezanson 等人，SIAM Rev.，59 (2017)，第 65-98 页]，而且有可能在代码中实现这种抽象。我们的实现展示了通用线性代数如何允许算子作为矩阵的元素。与 "运算符重载 "相比，"泛型编程 "通常需要重写反斜线才能利用运算符的优势，而 "泛型编程 "则没有这种必要。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

SIAM Journal on Matrix Analysis and Applications 数学-应用数学

CiteScore

2.90

自引率

6.70%

发文量

审稿时长

6-12 weeks

期刊介绍： The SIAM Journal on Matrix Analysis and Applications contains research articles in matrix analysis and its applications and papers of interest to the numerical linear algebra community. Applications include such areas as signal processing, systems and control theory, statistics, Markov chains, and mathematical biology. Also contains papers that are of a theoretical nature but have a possible impact on applications.