Neural network training with constrained integer weights

Abstract

Presents neural network training algorithms which are based on the differential evolution (DE) strategies introduced by Storn and Price (J. of Global Optimization, vol. 11, pp. 341-59, 1997). These strategies are applied to train neural networks with small integer weights. Such neural networks are better suited for hardware implementation than the real weight ones. Furthermore, we constrain the weights and biases in the range [-2/sup k/+1, 2/sup k/-1], for k=3,4,5. Thus, they can be represented by just k bits. These algorithms have been designed keeping in mind that the resulting integer weights require less bits to be stored and the digital arithmetic operations between them are more easily implemented in hardware. Obviously, if the network is trained in a constrained weight space, smaller weights are found and less memory is required. On the other hand, the network training procedure can be more effective and efficient when large weights are allowed. Thus, for a given application, a trade-off between effectiveness and memory consumption has to be considered. We present the results of evolution algorithms for this difficult task. Based on the application of the proposed class of methods on classical neural network benchmarks, our experience is that these methods are effective and reliable.