Applied and Computational Harmonic Analysis最新文献

{"title":"LU decomposition and Toeplitz decomposition of a neural network","authors":"Yucong Liu , Simiao Jiao , Lek-Heng Lim","doi":"10.1016/j.acha.2023.101601","DOIUrl":"https://doi.org/10.1016/j.acha.2023.101601","url":null,"abstract":"<div><p>Any matrix <em>A</em> has an LU decomposition up to a row or column permutation. Less well-known is the fact that it has a ‘Toeplitz decomposition’ <span><math><mi>A</mi><mo>=</mo><msub><mrow><mi>T</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi>T</mi></mrow><mrow><mn>2</mn></mrow></msub><mo>⋯</mo><msub><mrow><mi>T</mi></mrow><mrow><mi>r</mi></mrow></msub></math></span> where <span><math><msub><mrow><mi>T</mi></mrow><mrow><mi>i</mi></mrow></msub></math></span>'s are Toeplitz matrices. We will prove that any continuous function <span><math><mi>f</mi><mo>:</mo><msup><mrow><mi>R</mi></mrow><mrow><mi>n</mi></mrow></msup><mo>→</mo><msup><mrow><mi>R</mi></mrow><mrow><mi>m</mi></mrow></msup></math></span> has an approximation to arbitrary accuracy by a neural network that maps <span><math><mi>x</mi><mo>∈</mo><msup><mrow><mi>R</mi></mrow><mrow><mi>n</mi></mrow></msup></math></span> to <span><math><msub><mrow><mi>L</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi>σ</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi>U</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi>σ</mi></mrow><mrow><mn>2</mn></mrow></msub><msub><mrow><mi>L</mi></mrow><mrow><mn>2</mn></mrow></msub><msub><mrow><mi>σ</mi></mrow><mrow><mn>3</mn></mrow></msub><msub><mrow><mi>U</mi></mrow><mrow><mn>2</mn></mrow></msub><mo>⋯</mo><msub><mrow><mi>L</mi></mrow><mrow><mi>r</mi></mrow></msub><msub><mrow><mi>σ</mi></mrow><mrow><mn>2</mn><mi>r</mi><mo>−</mo><mn>1</mn></mrow></msub><msub><mrow><mi>U</mi></mrow><mrow><mi>r</mi></mrow></msub><mi>x</mi><mo>∈</mo><msup><mrow><mi>R</mi></mrow><mrow><mi>m</mi></mrow></msup></math></span>, i.e., where the weight matrices alternate between lower and upper triangular matrices, <span><math><msub><mrow><mi>σ</mi></mrow><mrow><mi>i</mi></mrow></msub><mo>(</mo><mi>x</mi><mo>)</mo><mo>≔</mo><mi>σ</mi><mo>(</mo><mi>x</mi><mo>−</mo><msub><mrow><mi>b</mi></mrow><mrow><mi>i</mi></mrow></msub><mo>)</mo></math></span> for some bias vector <span><math><msub><mrow><mi>b</mi></mrow><mrow><mi>i</mi></mrow></msub></math></span>, and the activation <em>σ</em> may be chosen to be essentially any uniformly continuous nonpolynomial function. The same result also holds with Toeplitz matrices, i.e., <span><math><mi>f</mi><mo>≈</mo><msub><mrow><mi>T</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi>σ</mi></mrow><mrow><mn>1</mn></mrow></msub><msub><mrow><mi>T</mi></mrow><mrow><mn>2</mn></mrow></msub><msub><mrow><mi>σ</mi></mrow><mrow><mn>2</mn></mrow></msub><mo>⋯</mo><msub><mrow><mi>σ</mi></mrow><mrow><mi>r</mi><mo>−</mo><mn>1</mn></mrow></msub><msub><mrow><mi>T</mi></mrow><mrow><mi>r</mi></mrow></msub></math></span> to arbitrary accuracy, and likewise for Hankel matrices. A consequence of our Toeplitz result is a fixed-width universal approximation theorem for convolutional neural networks, which so far have only arbitrary width versions. Since our results apply in particular to the case when <em>f</em> is a general neural network, we ma","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"68 ","pages":"Article 101601"},"PeriodicalIF":2.5,"publicationDate":"2023-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49778387","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Representation of operators using fusion frames","authors":"Peter Balazs,&nbsp;Mitra Shamsabadi,&nbsp;Ali Akbar Arefijamaal,&nbsp;Gilles Chardon","doi":"10.1016/j.acha.2023.101596","DOIUrl":"https://doi.org/10.1016/j.acha.2023.101596","url":null,"abstract":"<div><p><span><span>To solve operator equations numerically, matrix representations are employing bases or more recently frames. For finding the numerical solution of operator equations a decomposition in subspaces is needed in many applications. To combine those two approaches, it is necessary to extend the known methods of matrix representation to the utilization of fusion frames. In this paper, we investigate this representation of operators on a </span>Hilbert space </span><span><math><mi>H</mi></math></span><span><span> with Bessel<span> fusion sequences, fusion frames and fusion Riesz bases. Fusion frames can be considered as a frame-like family of subspaces. Taking the particular property of the duality of fusion frames into account, this allows us to define a matrix representation in a canonical as well as an alternate way, the later being more efficient and well behaved in respect to inversion. We will give the basic definitions and show some structural results, like that the functions assigning the alternate representation to an operator is an algebra </span></span>homomorphism. We give formulas for pseudo-inverses and the inverses (if existing) of such matrix representations. We apply this idea to Schatten </span><em>p</em><span>-class operators. Consequently, we show that tensor products of fusion frames are fusion frames in the space of Hilbert-Schmidt operators. We will show how this can be used for the solution of operator equations and link our approach to the additive Schwarz algorithm. Consequently, we propose some methods for solving an operator equation by iterative methods on the subspaces. Moreover, we implement the alternate Schwarz algorithms employing our perspective and provide small proof-of-concept numerical experiments. Finally, we show the application of this concept to overlapped convolution and the non-standard wavelet representation of operators.</span></p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"68 ","pages":"Article 101596"},"PeriodicalIF":2.5,"publicationDate":"2023-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49778388","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Diffusion maps for embedded manifolds with boundary with applications to PDEs","authors":"Ryan Vaughn ,&nbsp;Tyrus Berry ,&nbsp;Harbir Antil","doi":"10.1016/j.acha.2023.101593","DOIUrl":"https://doi.org/10.1016/j.acha.2023.101593","url":null,"abstract":"<div><p><span><span><span>Given only a finite collection of points sampled from a Riemannian manifold embedded in a </span>Euclidean space<span><span>, in this paper we propose a new method to numerically solve elliptic and parabolic partial differential equations (PDEs) supplemented with boundary conditions. Since the construction of triangulations on unknown manifolds can be both difficult and expensive, both in terms of computational and data requirements, our goal is to solve these problems without a triangulation. Instead, we rely only on using the sample points to define </span>quadrature formulas<span> on the unknown manifold. Our main tool is the diffusion maps algorithm. We re-analyze this well-known method in a variational sense for manifolds with boundary. Our main result is that the variational diffusion maps graph Laplacian<span> is a consistent estimator of the Dirichlet energy on the manifold. This improves upon previous results and provides a rigorous justification of the well-known relationship between diffusion maps and the Neumann </span></span></span></span>eigenvalue problem<span>. Moreover, using semigeodesic coordinates we derive the first uniform asymptotic expansion of the diffusion maps kernel integral operator for manifolds with boundary. This expansion relies on a novel lemma which relates the extrinsic </span></span>Euclidean distance to the coordinate norm in a normal collar of the boundary. We then use a recently developed method of estimating the distance to boundary function (notice that the boundary location is assumed to be unknown) to construct a consistent estimator for boundary integrals. Finally, by combining these various estimators, we illustrate how to impose Dirichlet and Neumann conditions for some common PDEs based on the Laplacian. Several numerical examples illustrate our theoretical findings.</p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"68 ","pages":"Article 101593"},"PeriodicalIF":2.5,"publicationDate":"2023-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49778390","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Metaplectic Gabor frames and symplectic analysis of time-frequency spaces","authors":"Elena Cordero ,&nbsp;Gianluca Giacchi","doi":"10.1016/j.acha.2023.101594","DOIUrl":"https://doi.org/10.1016/j.acha.2023.101594","url":null,"abstract":"<div><p>We introduce new frames, called <em>metaplectic Gabor frames</em><span>, as natural generalizations of Gabor frames in the framework of metaplectic Wigner distributions, cf. </span><span>[7]</span>, <span>[8]</span>, <span>[5]</span>, <span>[17]</span>, <span>[27]</span>, <span>[28]</span><span>. Namely, we develop the theory of metaplectic atoms in a full-general setting and prove an inversion formula for metaplectic Wigner distributions on </span><span><math><msup><mrow><mi>R</mi></mrow><mrow><mi>d</mi></mrow></msup></math></span><span>. Its discretization provides metaplectic Gabor frames.</span></p><p>Next, we deepen the understanding of the so-called shift-invertible metaplectic Wigner distributions, showing that they can be represented, up to chirps, as rescaled short-time Fourier transforms. As an application, we derive a new characterization of modulation and Wiener amalgam spaces. Thus, these metaplectic distributions (and related frames) provide meaningful definitions of local frequencies and can be used to measure effectively the local frequency content of signals.</p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"68 ","pages":"Article 101594"},"PeriodicalIF":2.5,"publicationDate":"2023-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49819289","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"On generalizations of the nonwindowed scattering transform","authors":"Albert Chua ,&nbsp;Matthew Hirn ,&nbsp;Anna Little","doi":"10.1016/j.acha.2023.101597","DOIUrl":"10.1016/j.acha.2023.101597","url":null,"abstract":"<div><p>In this paper, we generalize finite depth wavelet scattering transforms, which we formulate as <span><math><msup><mrow><mi>L</mi></mrow><mrow><mi>q</mi></mrow></msup><mo>(</mo><msup><mrow><mi>R</mi></mrow><mrow><mi>n</mi></mrow></msup><mo>)</mo></math></span><span> norms of a cascade of continuous wavelet transforms (or dyadic wavelet transforms) and contractive nonlinearities. We then provide norms for these operators, prove that these operators are well-defined, and are Lipschitz continuous to the action of </span><span><math><msup><mrow><mi>C</mi></mrow><mrow><mn>2</mn></mrow></msup></math></span><span> diffeomorphisms<span> in specific cases. Lastly, we extend our results to formulate an operator invariant to the action of rotations </span></span><span><math><mi>R</mi><mo>∈</mo><mtext>SO</mtext><mo>(</mo><mi>n</mi><mo>)</mo></math></span> and an operator that is equivariant to the action of rotations of <span><math><mi>R</mi><mo>∈</mo><mtext>SO</mtext><mo>(</mo><mi>n</mi><mo>)</mo></math></span>.</p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"68 ","pages":"Article 101597"},"PeriodicalIF":2.5,"publicationDate":"2023-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10552568/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41124001","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank","authors":"Hung-Hsu Chou ,&nbsp;Carsten Gieshoff ,&nbsp;Johannes Maly ,&nbsp;Holger Rauhut","doi":"10.1016/j.acha.2023.101595","DOIUrl":"https://doi.org/10.1016/j.acha.2023.101595","url":null,"abstract":"<div><p>In deep learning<span>, it is common to use more network parameters than training points. In such scenario of over-parameterization, there are usually multiple networks that achieve zero training error so that the training algorithm induces an implicit bias on the computed solution. In practice, (stochastic) gradient descent tends to prefer solutions which generalize well, which provides a possible explanation of the success of deep learning. In this paper we analyze the dynamics of gradient descent in the simplified setting of linear networks and of an estimation problem. Although we are not in an overparameterized scenario, our analysis nevertheless provides insights into the phenomenon of implicit bias. In fact, we derive a rigorous analysis of the dynamics of vanilla gradient descent, and characterize the dynamical convergence of the spectrum. We are able to accurately locate time intervals where the effective rank of the iterates is close to the effective rank of a low-rank projection of the ground-truth matrix. In practice, those intervals can be used as criteria for early stopping if a certain regularity is desired. We also provide empirical evidence for implicit bias in more general scenarios, such as matrix sensing and random initialization. This suggests that deep learning prefers trajectories whose complexity (measured in terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for the theoretical understanding of deep learning.</span></p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"68 ","pages":"Article 101595"},"PeriodicalIF":2.5,"publicationDate":"2023-09-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49778391","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Finite alphabet phase retrieval","authors":"Tamir Bendory,&nbsp;Dan Edidin,&nbsp;Ivan Gonzalez","doi":"10.1016/j.acha.2023.04.005","DOIUrl":"https://doi.org/10.1016/j.acha.2023.04.005","url":null,"abstract":"<div><p>We consider the finite alphabet phase retrieval problem: recovering a signal whose entries lie in a small alphabet of possible values from its Fourier magnitudes. This problem arises in the celebrated technology of X-ray crystallography to determine the atomic structure of biological molecules. Our main result states that for generic values of the alphabet, two signals have the same Fourier magnitudes if and only if several partitions have the same difference sets. Thus, the finite alphabet phase retrieval problem reduces to the combinatorial problem of determining a signal from those difference sets. Notably, this result holds true when one of the letters of the alphabet is zero, namely, for sparse signals with finite alphabet, which is the situation in X-ray crystallography.</p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"66 ","pages":"Pages 151-160"},"PeriodicalIF":2.5,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49726863","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Near-optimal bounds for generalized orthogonal Procrustes problem via generalized power method","authors":"Shuyang Ling","doi":"10.1016/j.acha.2023.04.008","DOIUrl":"https://doi.org/10.1016/j.acha.2023.04.008","url":null,"abstract":"<div><p><span><span>Given multiple point clouds, how to find the rigid transform (rotation, reflection, and shifting) such that these point clouds are well aligned? This problem, known as the generalized orthogonal Procrustes problem (GOPP), has found numerous applications in statistics<span>, computer vision, and imaging science. While one commonly-used method is finding the least squares estimator, it is generally an NP-hard problem to obtain the least squares estimator exactly due to the notorious nonconvexity. In this work, we apply the semidefinite programming (SDP) relaxation and the generalized power method to solve this generalized orthogonal Procrustes problem. In particular, we assume the data are generated from a signal-plus-noise model: each observed point cloud is a noisy copy of the same unknown point cloud transformed by an unknown </span></span>orthogonal matrix<span> and also corrupted by additive Gaussian noise. We show that the generalized power method (equivalently alternating minimization algorithm) with spectral initialization converges to the unique global optimum to the SDP relaxation, provided that the signal-to-noise ratio is high. Moreover, this limiting point is exactly the least squares estimator and also the maximum likelihood estimator. Our theoretical bound is near-optimal in terms of the information-theoretic limit (only loose by a factor of the dimension and a log factor). Our results significantly improve the state-of-the-art results on the tightness of the SDP relaxation for the generalized orthogonal Procrustes problem, an open problem posed by Bandeira et al. (2014) </span></span><span>[8]</span>.</p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"66 ","pages":"Pages 62-100"},"PeriodicalIF":2.5,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49726663","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Decentralized learning over a network with Nyström approximation using SGD","authors":"Heng Lian ,&nbsp;Jiamin Liu","doi":"10.1016/j.acha.2023.06.005","DOIUrl":"10.1016/j.acha.2023.06.005","url":null,"abstract":"<div><p>Nowadays we often meet with a learning problem when data are distributed on different machines connected via a network, instead of stored centrally. Here we consider decentralized supervised learning in a reproducing kernel Hilbert space<span>. We note that standard gradient descent in a reproducing kernel Hilbert space is difficult to implement with multiple communications between worker machines. On the other hand, the Nyström approximation using gradient descent is more suited for the decentralized setting since only a small number of data points need to be shared at the beginning of the algorithm. In the setting of decentralized distributed learning in a reproducing kernel Hilbert space, we establish the optimal learning rate of stochastic gradient descent based on mini-batches, allowing multiple passes over the data set. The proposal provides a scalable approach to nonparametric estimation combining gradient method, distributed estimation, and random projection.</span></p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"66 ","pages":"Pages 373-387"},"PeriodicalIF":2.5,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44914607","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PiPs: A kernel-based optimization scheme for analyzing non-stationary 1D signals","authors":"Jieren Xu ,&nbsp;Yitong Li ,&nbsp;Haizhao Yang ,&nbsp;David Dunson ,&nbsp;Ingrid Daubechies","doi":"10.1016/j.acha.2023.04.002","DOIUrl":"10.1016/j.acha.2023.04.002","url":null,"abstract":"<div><p>This paper proposes a novel kernel-based optimization scheme to handle tasks in the analysis, <em>e.g.</em>, signal spectral estimation and single-channel source separation of 1D non-stationary oscillatory data. The key insight of our optimization scheme for reconstructing the time-frequency information is that when a nonparametric regression is applied on some input values, the output regressed points would lie near the oscillatory pattern of the oscillatory 1D signal only if these input values are a good approximation of the ground-truth phase function. In this work, <em>Gaussian Process (GP)</em> is chosen to conduct this nonparametric regression: the oscillatory pattern is encoded as the <em>Pattern-inducing Points (PiPs)</em> which act as the training data points in the GP regression; while the targeted phase function is fed in to compute the correlation kernels, acting as the testing input. Better approximated phase function generates more precise kernels, thus resulting in smaller optimization loss error when comparing the kernel-based regression output with the original signals. To the best of our knowledge, this is the first algorithm that can satisfactorily handle fully non-stationary oscillatory data, close and crossover frequencies, and general oscillatory patterns. Even in the example of a signal produced by slow variation in the parameters of a trigonometric expansion, we show that PiPs admits competitive or better performance in terms of accuracy and robustness than existing state-of-the-art algorithms.</p></div>","PeriodicalId":55504,"journal":{"name":"Applied and Computational Harmonic Analysis","volume":"66 ","pages":"Pages 1-17"},"PeriodicalIF":2.5,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43227419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}