An algorithm for in-the-loop training based on activation function derivative approximation

Abstract

In this paper, we propose an algorithm for the in-the-loop training of a VLSI implementation of a neural network with analog neurons and programmable digital weights. The difficulty in evaluating the derivative of nonideal activation functions has been the main obstacle to effectively training a VLSI neural network chip via the standard backpropagation (BP) algorithm. In the paper approximated derivatives have been used in BP algorithm incorporating an adaptive learning rate. An analysis from the viewpoint of optimization shows the proposed algorithm is advantageous. Experimental results indicate that the algorithm is superior to weight perturbation-based algorithms.