高维多指数模型中弱可学性的基本限制

Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, Florent Krzakala
{"title":"高维多指数模型中弱可学性的基本限制","authors":"Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, Florent Krzakala","doi":"arxiv-2405.15480","DOIUrl":null,"url":null,"abstract":"Multi-index models -- functions which only depend on the covariates through a\nnon-linear transformation of their projection on a subspace -- are a useful\nbenchmark for investigating feature learning with neural networks. This paper\nexamines the theoretical boundaries of learnability in this hypothesis class,\nfocusing particularly on the minimum sample complexity required for weakly\nrecovering their low-dimensional structure with first-order iterative\nalgorithms, in the high-dimensional regime where the number of samples is\n$n=\\alpha d$ is proportional to the covariate dimension $d$. Our findings\nunfold in three parts: (i) first, we identify under which conditions a\n\\textit{trivial subspace} can be learned with a single step of a first-order\nalgorithm for any $\\alpha\\!>\\!0$; (ii) second, in the case where the trivial\nsubspace is empty, we provide necessary and sufficient conditions for the\nexistence of an {\\it easy subspace} consisting of directions that can be\nlearned only above a certain sample complexity $\\alpha\\!>\\!\\alpha_c$. The\ncritical threshold $\\alpha_{c}$ marks the presence of a computational phase\ntransition, in the sense that no efficient iterative algorithm can succeed for\n$\\alpha\\!<\\!\\alpha_c$. In a limited but interesting set of really hard\ndirections -- akin to the parity problem -- $\\alpha_c$ is found to diverge.\nFinally, (iii) we demonstrate that interactions between different directions\ncan result in an intricate hierarchical learning phenomenon, where some\ndirections can be learned sequentially when coupled to easier ones. Our\nanalytical approach is built on the optimality of approximate message-passing\nalgorithms among first-order iterative methods, delineating the fundamental\nlearnability limit across a broad spectrum of algorithms, including neural\nnetworks trained with gradient descent.","PeriodicalId":501066,"journal":{"name":"arXiv - PHYS - Disordered Systems and Neural Networks","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-05-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Fundamental limits of weak learnability in high-dimensional multi-index models\",\"authors\":\"Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, Florent Krzakala\",\"doi\":\"arxiv-2405.15480\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Multi-index models -- functions which only depend on the covariates through a\\nnon-linear transformation of their projection on a subspace -- are a useful\\nbenchmark for investigating feature learning with neural networks. This paper\\nexamines the theoretical boundaries of learnability in this hypothesis class,\\nfocusing particularly on the minimum sample complexity required for weakly\\nrecovering their low-dimensional structure with first-order iterative\\nalgorithms, in the high-dimensional regime where the number of samples is\\n$n=\\\\alpha d$ is proportional to the covariate dimension $d$. Our findings\\nunfold in three parts: (i) first, we identify under which conditions a\\n\\\\textit{trivial subspace} can be learned with a single step of a first-order\\nalgorithm for any $\\\\alpha\\\\!>\\\\!0$; (ii) second, in the case where the trivial\\nsubspace is empty, we provide necessary and sufficient conditions for the\\nexistence of an {\\\\it easy subspace} consisting of directions that can be\\nlearned only above a certain sample complexity $\\\\alpha\\\\!>\\\\!\\\\alpha_c$. The\\ncritical threshold $\\\\alpha_{c}$ marks the presence of a computational phase\\ntransition, in the sense that no efficient iterative algorithm can succeed for\\n$\\\\alpha\\\\!<\\\\!\\\\alpha_c$. In a limited but interesting set of really hard\\ndirections -- akin to the parity problem -- $\\\\alpha_c$ is found to diverge.\\nFinally, (iii) we demonstrate that interactions between different directions\\ncan result in an intricate hierarchical learning phenomenon, where some\\ndirections can be learned sequentially when coupled to easier ones. Our\\nanalytical approach is built on the optimality of approximate message-passing\\nalgorithms among first-order iterative methods, delineating the fundamental\\nlearnability limit across a broad spectrum of algorithms, including neural\\nnetworks trained with gradient descent.\",\"PeriodicalId\":501066,\"journal\":{\"name\":\"arXiv - PHYS - Disordered Systems and Neural Networks\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-05-24\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - PHYS - Disordered Systems and Neural Networks\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2405.15480\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - PHYS - Disordered Systems and Neural Networks","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2405.15480","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

多指数模型是研究神经网络特征学习的一个有用基准,多指数模型是指只通过子空间投影的非线性变换依赖于协变量的函数。本论文研究了这一假设类别的可学习性理论边界,尤其关注在样本数为$n=α d$与协变量维度$d$成正比的高维条件下,用一阶迭代算法弱恢复其低维结构所需的最小样本复杂度。我们的发现分为三个部分:(i)首先,我们确定了在哪些条件下,对于任意 $\alpha\!>\!0$ 的一阶算法可以通过一步学习到一个{textit{trivial子空间};(ii)其次,在trivial子空间为空的情况下,我们为{it easy子空间}的存在提供了必要条件和充分条件,这个{it easy子空间}由只能在一定的样本复杂度 $\alpha\!>\!\alpha_c$ 以上才能学习到的方向组成。临界阈值$\alpha_{c}$标志着计算阶段性转换的存在,从这个意义上说,对于$\alpha!<\!\alpha_c$,任何高效的迭代算法都无法成功。最后,(iii) 我们证明了不同方向之间的相互作用会导致一种错综复杂的分层学习现象,在这种现象中,当某些方向与更容易的方向耦合在一起时,它们可以被连续地学习。我们的分析方法建立在一阶迭代法中近似消息传递算法的最优性基础上,划定了包括用梯度下降训练的神经网络在内的各种算法的基本可学习极限。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Fundamental limits of weak learnability in high-dimensional multi-index models
Multi-index models -- functions which only depend on the covariates through a non-linear transformation of their projection on a subspace -- are a useful benchmark for investigating feature learning with neural networks. This paper examines the theoretical boundaries of learnability in this hypothesis class, focusing particularly on the minimum sample complexity required for weakly recovering their low-dimensional structure with first-order iterative algorithms, in the high-dimensional regime where the number of samples is $n=\alpha d$ is proportional to the covariate dimension $d$. Our findings unfold in three parts: (i) first, we identify under which conditions a \textit{trivial subspace} can be learned with a single step of a first-order algorithm for any $\alpha\!>\!0$; (ii) second, in the case where the trivial subspace is empty, we provide necessary and sufficient conditions for the existence of an {\it easy subspace} consisting of directions that can be learned only above a certain sample complexity $\alpha\!>\!\alpha_c$. The critical threshold $\alpha_{c}$ marks the presence of a computational phase transition, in the sense that no efficient iterative algorithm can succeed for $\alpha\!<\!\alpha_c$. In a limited but interesting set of really hard directions -- akin to the parity problem -- $\alpha_c$ is found to diverge. Finally, (iii) we demonstrate that interactions between different directions can result in an intricate hierarchical learning phenomenon, where some directions can be learned sequentially when coupled to easier ones. Our analytical approach is built on the optimality of approximate message-passing algorithms among first-order iterative methods, delineating the fundamental learnability limit across a broad spectrum of algorithms, including neural networks trained with gradient descent.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信