LU decomposition and Toeplitz decomposition of a neural network

Abstract

Any matrix A has an LU decomposition up to a row or column permutation. Less well-known is the fact that it has a ‘Toeplitz decomposition’ $A=T_1{T}_2\cdots T_r$ where $T_i$'s are Toeplitz matrices. We will prove that any continuous function $f:R^n\to R^m$ has an approximation to arbitrary accuracy by a neural network that maps $x\in R^n$ to $L_1{\sigma }_1{U}_1{\sigma }_2{L}_2{\sigma }_3{U}_2\cdots L_r{\sigma }_{2r-1}{U}_{r}x\in R^m$, i.e., where the weight matrices alternate between lower and upper triangular matrices, ${\sigma }_i\left(x\right)≔\sigma (x-b_i)$ for some bias vector $b_i$, and the activation σ may be chosen to be essentially any uniformly continuous nonpolynomial function. The same result also holds with Toeplitz matrices, i.e., $f\approx T_1{\sigma }_1{T}_2{\sigma }_2\cdots {\sigma }_{r-1}{T}_r$ to arbitrary accuracy, and likewise for Hankel matrices. A consequence of our Toeplitz result is a fixed-width universal approximation theorem for convolutional neural networks, which so far have only arbitrary width versions. Since our results apply in particular to the case when f is a general neural network, we may regard them as LU and Toeplitz decompositions of a neural network. The practical implication of our results is that one may vastly reduce the number of weight parameters in a neural network without sacrificing its power of universal approximation. We will present several experiments on real data sets to show that imposing such structures on the weight matrices dramatically reduces the number of training parameters with almost no noticeable effect on test accuracy.