Optimally generalizing neural networks

Abstract

The problem of approximating a real function f of L variables, given only in terms of its values y/sub 1/,. . .,y/sub M/ at a small set of sample points x/sub 1/,. . .,x/sub M/ in R/sup L/, is studied in the context of multilayer neural networks. Using the theory of reproducing kernels of Hilbert spaces, it is shown that this problem is the inverse of a linear model relating the values y/sub m/ to the function f itself. The authors consider the least-mean-square training criterion for nonlinear multilayer neural network architectures that learn the training set completely. The generalization property of a neural network is defined in terms of function reconstruction and the concept of the optimally generalizing neural network (OGNN) is proposed. It is a network that minimizes a criterion given in terms of the true error between the original function f and the reconstruction f/sub 1/ in the function space, instead of minimizing the error at the sample points only. As an example of the OGNN, a projection filter (PF) criterion is considered and the PFGNN is introduced. The network is of the two-layer nonlinear-linear type.<>