Stochastic approximation techniques and associated tools for neural network optimization

Abstract

This paper is devoted to the optimization of feedforward and feedback artificial neural networks (ANN) working in supervised learning mode. We describe in a general way how it is possible to derive first and second order stochastic approximation methods that provide learning capabilities. We show how certain variables, the sensitivities of the ANN outputs, play a key role in the ANN optimization process. Then we describe how some useful and elementary tools known in circuit theory can be used to compute these sensitivities with a low computational cost. We show on an example how to apply these two sets of complementary tools, i.e. stochastic approximation and sensitivity theory.