Relationship between fault tolerance, generalization and the Vapnik-Chervonenkis (VC) dimension of feedforward ANNs

Abstract

It is demonstrated that fault tolerance, generalization and the Vapnik-Chertonenkis (VC) dimension are inter-related attributes. It is well known that the generalization error if plotted as a function of the VC dimension h, exhibits a well defined minimum corresponding to an optimal value of h, say h/sub opt/. We show that if the VC dimension h of an ANN satisfies h/spl les/h/sub opt/ (i.e., there is no excess capacity or redundancy), then fault tolerance and generalization are mutually conflicting attributes. On the other hand, if h>h/sub opt/ (i.e., there is excess capacity or redundancy), then fault tolerance and generalization are mutually synergistic attributes. In other words, training methods geared towards improving the fault tolerance can also lead to better generalization and vice versa, only when there is excess capacity or redundancy. This is consistent with our previous results indicating that complete fault tolerance in ANNs requires a significant amount of redundancy.