Efficient identification of wide shallow neural networks with biases

IF 2.6 2区 数学 Q1 MATHEMATICS, APPLIED
Massimo Fornasier , Timo Klock , Marco Mondelli , Michael Rauchensteiner
{"title":"Efficient identification of wide shallow neural networks with biases","authors":"Massimo Fornasier ,&nbsp;Timo Klock ,&nbsp;Marco Mondelli ,&nbsp;Michael Rauchensteiner","doi":"10.1016/j.acha.2025.101749","DOIUrl":null,"url":null,"abstract":"<div><div>The identification of the parameters of a neural network from finite samples of input-output pairs is often referred to as the <em>teacher-student model</em>, and this model has represented a popular framework for understanding training and generalization. Even if the problem is NP-complete in the worst case, a rapidly growing literature – after adding suitable distributional assumptions – has established finite sample identification of two-layer networks with a number of neurons <span><math><mi>m</mi><mo>=</mo><mi>O</mi><mo>(</mo><mi>D</mi><mo>)</mo></math></span>, <em>D</em> being the input dimension. For the range <span><math><mi>D</mi><mo>&lt;</mo><mi>m</mi><mo>&lt;</mo><msup><mrow><mi>D</mi></mrow><mrow><mn>2</mn></mrow></msup></math></span> the problem becomes harder, and truly little is known for networks parametrized by biases as well. This paper fills the gap by providing efficient algorithms and rigorous theoretical guarantees of finite sample identification for such wider shallow networks with biases. Our approach is based on a two-step pipeline: first, we recover the direction of the weights, by exploiting second order information; next, we identify the signs by suitable algebraic evaluations, and we recover the biases by empirical risk minimization via gradient descent. Numerical results demonstrate the effectiveness of our approach.</div></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"77 ","pages":"Article 101749"},"PeriodicalIF":2.6000,"publicationDate":"2025-02-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Applied and Computational Harmonic Analysis","FirstCategoryId":"100","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S106352032500003X","RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MATHEMATICS, APPLIED","Score":null,"Total":0}
引用次数: 0

Abstract

The identification of the parameters of a neural network from finite samples of input-output pairs is often referred to as the teacher-student model, and this model has represented a popular framework for understanding training and generalization. Even if the problem is NP-complete in the worst case, a rapidly growing literature – after adding suitable distributional assumptions – has established finite sample identification of two-layer networks with a number of neurons m=O(D), D being the input dimension. For the range D<m<D2 the problem becomes harder, and truly little is known for networks parametrized by biases as well. This paper fills the gap by providing efficient algorithms and rigorous theoretical guarantees of finite sample identification for such wider shallow networks with biases. Our approach is based on a two-step pipeline: first, we recover the direction of the weights, by exploiting second order information; next, we identify the signs by suitable algebraic evaluations, and we recover the biases by empirical risk minimization via gradient descent. Numerical results demonstrate the effectiveness of our approach.
带偏差的宽浅层神经网络的有效识别
从有限的输入输出对样本中识别神经网络的参数通常被称为师生模型,这种模型代表了一种理解训练和泛化的流行框架。即使在最坏的情况下问题是np完全的,在添加合适的分布假设之后,快速增长的文献已经建立了具有若干神经元m=O(D)的两层网络的有限样本识别,D是输入维。对于D<;m<;D2范围,问题变得更加困难,对于被偏差参数化的网络,我们也知之甚少。本文通过提供有效的算法和严格的理论保证来填补这一空白,用于这种具有偏差的更广泛的浅层网络的有限样本识别。我们的方法基于两步管道:首先,我们通过利用二阶信息恢复权重的方向;接下来,我们通过适当的代数评估来识别符号,并通过梯度下降的经验风险最小化来恢复偏差。数值结果证明了该方法的有效性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Applied and Computational Harmonic Analysis
Applied and Computational Harmonic Analysis 物理-物理:数学物理
CiteScore
5.40
自引率
4.00%
发文量
67
审稿时长
22.9 weeks
期刊介绍: Applied and Computational Harmonic Analysis (ACHA) is an interdisciplinary journal that publishes high-quality papers in all areas of mathematical sciences related to the applied and computational aspects of harmonic analysis, with special emphasis on innovative theoretical development, methods, and algorithms, for information processing, manipulation, understanding, and so forth. The objectives of the journal are to chronicle the important publications in the rapidly growing field of data representation and analysis, to stimulate research in relevant interdisciplinary areas, and to provide a common link among mathematical, physical, and life scientists, as well as engineers.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信