Journal of Complexity最新文献

{"title":"Rates of approximation by ReLU shallow neural networks","authors":"Tong Mao ,&nbsp;Ding-Xuan Zhou","doi":"10.1016/j.jco.2023.101784","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101784","url":null,"abstract":"<div><p>Neural networks activated by the rectified linear unit (ReLU) play a central role in the recent development of deep learning. The topic of approximating functions from Hölder spaces by these networks is crucial for understanding the efficiency of the induced learning algorithms. Although the topic has been well investigated in the setting of deep neural networks with many layers of hidden neurons, it is still open for shallow networks having only one hidden layer. In this paper, we provide rates of uniform approximation by these networks. We show that ReLU shallow neural networks with <em>m</em> hidden neurons can uniformly approximate functions from the Hölder space <span><math><msubsup><mrow><mi>W</mi></mrow><mrow><mo>∞</mo></mrow><mrow><mi>r</mi></mrow></msubsup><mo>(</mo><msup><mrow><mo>[</mo><mo>−</mo><mn>1</mn><mo>,</mo><mn>1</mn><mo>]</mo></mrow><mrow><mi>d</mi></mrow></msup><mo>)</mo></math></span> with rates <span><math><mi>O</mi><mo>(</mo><msup><mrow><mo>(</mo><mi>log</mi><mo>⁡</mo><mi>m</mi><mo>)</mo></mrow><mrow><mfrac><mrow><mn>1</mn></mrow><mrow><mn>2</mn></mrow></mfrac><mo>+</mo><mi>d</mi></mrow></msup><msup><mrow><mi>m</mi></mrow><mrow><mo>−</mo><mfrac><mrow><mi>r</mi></mrow><mrow><mi>d</mi></mrow></mfrac><mfrac><mrow><mi>d</mi><mo>+</mo><mn>2</mn></mrow><mrow><mi>d</mi><mo>+</mo><mn>4</mn></mrow></mfrac></mrow></msup><mo>)</mo></math></span> when <span><math><mi>r</mi><mo>&lt;</mo><mi>d</mi><mo>/</mo><mn>2</mn><mo>+</mo><mn>2</mn></math></span>. Such rates are very close to the optimal one <span><math><mi>O</mi><mo>(</mo><msup><mrow><mi>m</mi></mrow><mrow><mo>−</mo><mfrac><mrow><mi>r</mi></mrow><mrow><mi>d</mi></mrow></mfrac></mrow></msup><mo>)</mo></math></span> in the sense that <span><math><mfrac><mrow><mi>d</mi><mo>+</mo><mn>2</mn></mrow><mrow><mi>d</mi><mo>+</mo><mn>4</mn></mrow></mfrac></math></span> is close to 1, when the dimension <em>d</em> is large.</p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101784"},"PeriodicalIF":1.7,"publicationDate":"2023-07-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49876977","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation","authors":"Xia Liu","doi":"10.1016/j.jco.2023.101783","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101783","url":null,"abstract":"<div><p><span><span>Constructing neural networks for function approximation is a classical and longstanding topic in </span>approximation theory. In this paper, we aim at constructing </span>deep neural networks with three hidden layers using a sigmoidal activation function to approximate smooth and sparse functions. Specifically, we prove that the constructed deep nets with controllable magnitude of free parameters can reach the optimal approximation rate in approximating both smooth and sparse functions. In particular, we prove that neural networks with three hidden layers can avoid the phenomenon of saturation, i.e., the phenomenon that for some neural network architectures, the approximation rate stops improving for functions of very high smoothness.</p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101783"},"PeriodicalIF":1.7,"publicationDate":"2023-07-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49876980","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Worst case tractability of linear problems in the presence of noise: Linear information","authors":"Leszek Plaskota,&nbsp;Paweł Siedlecki","doi":"10.1016/j.jco.2023.101782","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101782","url":null,"abstract":"<div><p><span>We study the worst case tractability of multivariate linear problems defined on separable Hilbert spaces. Information about a problem instance consists of noisy evaluations of arbitrary bounded </span>linear functionals, where the noise is either deterministic or random. The cost of a single evaluation depends on its precision and is controlled by a cost function. We establish mutual interactions between tractability of a problem with noisy information, the cost function, and tractability of the same problem, but with exact information.</p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101782"},"PeriodicalIF":1.7,"publicationDate":"2023-07-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49876976","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On the complexity of a unified convergence analysis for iterative methods","authors":"Ioannis K. Argyros ,&nbsp;Stepan Shakhno ,&nbsp;Samundra Regmi ,&nbsp;Halyna Yarmola","doi":"10.1016/j.jco.2023.101781","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101781","url":null,"abstract":"<div><p><span><span>A local and a semi-local convergence of general iterative methods for solving nonlinear operator equations in </span>Banach spaces is developed under </span><em>ω</em>-continuity conditions. Our approach unifies existing results and provides a new way of studying iterative methods. The main idea is to find a more accurate domain containing the iterates. No extra effort is used to obtain this. Also, the results of the numerical experiments are given that confirm obtained theoretical estimates.</p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101781"},"PeriodicalIF":1.7,"publicationDate":"2023-07-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49876979","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation","authors":"Xia Liu","doi":"10.1016/j.jco.2023.101783","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101783","url":null,"abstract":"","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 1","pages":"101783"},"PeriodicalIF":1.7,"publicationDate":"2023-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"54746300","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The rate of convergence for sparse and low-rank quantile trace regression","authors":"Xiangyong Tan ,&nbsp;Ling Peng ,&nbsp;Peiwen Xiao ,&nbsp;Qing Liu ,&nbsp;Xiaohui Liu","doi":"10.1016/j.jco.2023.101778","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101778","url":null,"abstract":"<div><p>Trace regression models are widely used in applications involving panel data, images, genomic microarrays, etc., where high-dimensional covariates<span> are often involved. However, the existing research involving high-dimensional covariates focuses mainly on the condition mean model. In this paper, we extend the trace regression model to the quantile trace regression model when the parameter is a matrix of simultaneously low rank and row (column) sparsity. The convergence rate of the penalized estimator is derived under mild conditions. Simulations, as well as a real data application, are also carried out for illustration.</span></p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101778"},"PeriodicalIF":1.7,"publicationDate":"2023-06-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49876978","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Optimal recovery and volume estimates","authors":"Alexander Kushpel","doi":"10.1016/j.jco.2023.101780","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101780","url":null,"abstract":"<div><p>We study volumes of sections of convex origin-symmetric bodies in <span><math><msup><mrow><mi>R</mi></mrow><mrow><mi>n</mi></mrow></msup></math></span><span> induced by orthonormal systems on probability spaces. The approach is based on volume estimates of John-Löwner ellipsoids and expectations of norms induced by the respective systems. The estimates obtained allow us to establish lower bounds for the radii of sections which gives lower bounds for Gelfand widths (or linear cowidths). As an application we offer a new method of evaluation of Gelfand and Kolmogorov widths of multiplier operators. In particular, we establish sharp orders of widths of standard Sobolev classes </span><span><math><msubsup><mrow><mi>W</mi></mrow><mrow><mi>p</mi></mrow><mrow><mi>γ</mi></mrow></msubsup></math></span>, <span><math><mi>γ</mi><mo>&gt;</mo><mn>0</mn></math></span> in <span><math><msub><mrow><mi>L</mi></mrow><mrow><mi>q</mi></mrow></msub></math></span> on two-point homogeneous spaces in the difficult case, i.e. if <span><math><mn>1</mn><mo>&lt;</mo><mi>q</mi><mo>≤</mo><mi>p</mi><mo>≤</mo><mo>∞</mo></math></span>.</p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101780"},"PeriodicalIF":1.7,"publicationDate":"2023-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49877027","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs","authors":"Dinh Dũng ,&nbsp;Van Kien Nguyen ,&nbsp;Duong Thanh Pham","doi":"10.1016/j.jco.2023.101779","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101779","url":null,"abstract":"<div><p><span><span>We investigate non-adaptive methods of deep ReLU neural network approximation in </span>Bochner spaces </span><span><math><msub><mrow><mi>L</mi></mrow><mrow><mn>2</mn></mrow></msub><mo>(</mo><msup><mrow><mi>U</mi></mrow><mrow><mo>∞</mo></mrow></msup><mo>,</mo><mi>X</mi><mo>,</mo><mi>μ</mi><mo>)</mo></math></span> of functions on <span><math><msup><mrow><mi>U</mi></mrow><mrow><mo>∞</mo></mrow></msup></math></span><span> taking values in a separable Hilbert space </span><em>X</em>, where <span><math><msup><mrow><mi>U</mi></mrow><mrow><mo>∞</mo></mrow></msup></math></span> is either <span><math><msup><mrow><mi>R</mi></mrow><mrow><mo>∞</mo></mrow></msup></math></span><span> equipped with the standard Gaussian probability measure, or </span><span><math><msup><mrow><mo>[</mo><mo>−</mo><mn>1</mn><mo>,</mo><mn>1</mn><mo>]</mo></mrow><mrow><mo>∞</mo></mrow></msup></math></span> equipped with the Jacobi probability measure. Functions to be approximated are assumed to satisfy a certain weighted <span><math><msub><mrow><mi>ℓ</mi></mrow><mrow><mn>2</mn></mrow></msub></math></span>-summability of the generalized chaos polynomial expansion coefficients with respect to the measure <em>μ</em><span>. We prove the convergence rate of this approximation in terms of the size of approximating deep ReLU neural networks. These results then are applied to approximation of the solution to parametric<span> elliptic PDEs with random inputs for the lognormal and affine cases.</span></span></p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101779"},"PeriodicalIF":1.7,"publicationDate":"2023-06-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49877026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The curse of dimensionality for the Lp-discrepancy with finite p","authors":"Erich Novak ,&nbsp;Friedrich Pillichshammer","doi":"10.1016/j.jco.2023.101769","DOIUrl":"https://doi.org/10.1016/j.jco.2023.101769","url":null,"abstract":"<div><p>The <span><math><msub><mrow><mi>L</mi></mrow><mrow><mi>p</mi></mrow></msub></math></span>-discrepancy is a quantitative measure for the irregularity of distribution of an <em>N</em>-element point set in the <em>d</em>-dimensional unit-cube, which is closely related to the worst-case error of quasi-Monte Carlo algorithms for numerical integration. It's inverse for dimension <em>d</em> and error threshold <span><math><mi>ε</mi><mo>∈</mo><mo>(</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>)</mo></math></span> is the minimal number of points in <span><math><msup><mrow><mo>[</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>)</mo></mrow><mrow><mi>d</mi></mrow></msup></math></span> such that the minimal normalized <span><math><msub><mrow><mi>L</mi></mrow><mrow><mi>p</mi></mrow></msub></math></span>-discrepancy is less or equal <em>ε</em>. It is well known, that the inverse of <span><math><msub><mrow><mi>L</mi></mrow><mrow><mn>2</mn></mrow></msub></math></span>-discrepancy grows exponentially fast with the dimension <em>d</em>, i.e., we have the curse of dimensionality, whereas the inverse of <span><math><msub><mrow><mi>L</mi></mrow><mrow><mo>∞</mo></mrow></msub></math></span>-discrepancy depends exactly linearly on <em>d</em>. The behavior of inverse of <span><math><msub><mrow><mi>L</mi></mrow><mrow><mi>p</mi></mrow></msub></math></span>-discrepancy for general <span><math><mi>p</mi><mo>∉</mo><mo>{</mo><mn>2</mn><mo>,</mo><mo>∞</mo><mo>}</mo></math></span> has been an open problem for many years. In this paper we show that the <span><math><msub><mrow><mi>L</mi></mrow><mrow><mi>p</mi></mrow></msub></math></span>-discrepancy suffers from the curse of dimensionality for all <em>p</em> in <span><math><mo>(</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>]</mo></math></span> which are of the form <span><math><mi>p</mi><mo>=</mo><mn>2</mn><mi>ℓ</mi><mo>/</mo><mo>(</mo><mn>2</mn><mi>ℓ</mi><mo>−</mo><mn>1</mn><mo>)</mo></math></span> with <span><math><mi>ℓ</mi><mo>∈</mo><mi>N</mi></math></span>.</p><p>This result follows from a more general result that we show for the worst-case error of numerical integration in an anchored Sobolev space with anchor 0 of once differentiable functions in each variable whose first derivative has finite <span><math><msub><mrow><mi>L</mi></mrow><mrow><mi>q</mi></mrow></msub></math></span>-norm, where <em>q</em> is an even positive integer satisfying <span><math><mn>1</mn><mo>/</mo><mi>p</mi><mo>+</mo><mn>1</mn><mo>/</mo><mi>q</mi><mo>=</mo><mn>1</mn></math></span>.</p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"79 ","pages":"Article 101769"},"PeriodicalIF":1.7,"publicationDate":"2023-06-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49877025","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On the cardinality of lower sets and universal discretization","authors":"F. Dai ,&nbsp;A. Prymak ,&nbsp;A. Shadrin ,&nbsp;V.N. Temlyakov ,&nbsp;S. Tikhonov","doi":"10.1016/j.jco.2022.101726","DOIUrl":"https://doi.org/10.1016/j.jco.2022.101726","url":null,"abstract":"<div><p>A set <em>Q</em> in <span><math><msubsup><mrow><mi>Z</mi></mrow><mrow><mo>+</mo></mrow><mrow><mi>d</mi></mrow></msubsup></math></span> is a lower set if <span><math><mo>(</mo><msub><mrow><mi>k</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mo>…</mo><mo>,</mo><msub><mrow><mi>k</mi></mrow><mrow><mi>d</mi></mrow></msub><mo>)</mo><mo>∈</mo><mi>Q</mi></math></span> implies <span><math><mo>(</mo><msub><mrow><mi>l</mi></mrow><mrow><mn>1</mn></mrow></msub><mo>,</mo><mo>…</mo><mo>,</mo><msub><mrow><mi>l</mi></mrow><mrow><mi>d</mi></mrow></msub><mo>)</mo><mo>∈</mo><mi>Q</mi></math></span> whenever <span><math><mn>0</mn><mo>≤</mo><msub><mrow><mi>l</mi></mrow><mrow><mi>i</mi></mrow></msub><mo>≤</mo><msub><mrow><mi>k</mi></mrow><mrow><mi>i</mi></mrow></msub></math></span> for all <em>i</em>. We derive new and refine known results regarding the cardinality of the lower sets of size <em>n</em> in <span><math><msubsup><mrow><mi>Z</mi></mrow><mrow><mo>+</mo></mrow><mrow><mi>d</mi></mrow></msubsup></math></span><span>. Next we apply these results for universal discretization of the </span><span><math><msub><mrow><mi>L</mi></mrow><mrow><mn>2</mn></mrow></msub></math></span>-norm of elements from <em>n</em><span>-dimensional subspaces of trigonometric polynomials generated by lower sets.</span></p></div>","PeriodicalId":50227,"journal":{"name":"Journal of Complexity","volume":"76 ","pages":"Article 101726"},"PeriodicalIF":1.7,"publicationDate":"2023-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49882271","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}