The Computational Capacity of Neurons with Binary Weights or, Why it is OK to be a Bit Neuron

Abstract

synaptic weights prescribed in the algorithms might, at least for some applications, be an artifact of the algorithms used. Practical considerations in the building of hardware for these networks also dictate the study of the effect of imposing limited dynamic ranges on coefficients on computational capacity and learning in these neural network models. There are, hence, cogent theoretical and practical spurs to study networks with dynamic range limited synapses. We investigate two fundamental issues: Can we compute efficiently with these networks? Can constrained networks learn? Setting up a simple network model with binary weights, we argue that there is very little loss in eschewing real interconnections in favour of binary links. We demonstrate rigorously that the computational capacity scales gracefully when synaptic dynamic range is reduced from the continuum to a single bit: with binary connections the achievable capacity is as much as half that with real interconnections. Analogous results appear to hold for learning within the constraints of binary interconnections. Convergence rates for binary learning are reduced, but there is qualitative similarity to learning performance without constraints. While the actual mathematical demonstrations are quire involved, the algorithms themselves are quite simple and appeal persuasively to intuition. Based in part on the thesis that it is arguably easier to implement binary links than real interconnections, researchers have been building small prototype networks which appear to function reasonably we1L3 Our results appear to provide theoretical support for such ventures. It may be possible to generalise the results to other situations involving distributed