Improving generalization performance by information minimization

Abstract

In this paper, we attempt to show that the information stored in networks must be as small as possible for the improvement of the generalization performance under the condition that the networks can produce targets with appropriate accuracy. The information is defined by the difference between maximum entropy or uncertainty and observed entropy. Borrowing a definition of fuzzy entropy, the uncertainty function is defined for the internal representation and represented by the equation: -/spl upsi//sub i/ log /spl upsi//sub i/-(1-/spl upsi//sub i/) log (1-/spl upsi//sub i/), where /spl upsi//sub i/ is a hidden unit activity. After having formulated an update rule for the minimization of the information, we applied the method to a problem of language acquisition: the inference of the past tense forms of regular verbs. Experimental results confirmed that by our method, the information was significantly decreased and the generalization performance was greatly improved.<>