Dissipation in neuromorphic computing: Fundamental bounds for feedforward networks

Abstract

We present the fundamental lower bound on dissipation in feedforward neural networks associated with the combined cost of the training and testing phases. Finite state automata descriptions of output generation and the weight updates during training, are used to derive the corresponding lower bounds in a physically grounded manner. The results are illustrated using a simple perceptron learning the AND classification task. The effects of the learning rate parameter and input probability distribution on the cost of dissipation are studied. Derivation of neural network learning algorithms that minimize the total dissipation cost of training are explored.