DALTON - Deep Local Learning in SNNs via Local Weights and Surrogate-Derivative Transfer

Abstract

Direct training of Spiking Neural Networks (SNNs) is a challenging task because of their inherent temporality. Added to it, the vanilla Back-propagation based methods are not applicable either, due to the non-differentiability of the spikes in SNNs. Surrogate-Derivative based methods with Back-propagation Through Time (BPTT) address these direct training challenges quite well; however, such methods are not neuromorphic-hardware friendly for the On-chip training of SNNs. Recently formalized Three-Factor based Rules (TFR) for direct local-training of SNNs are neuromorphic-hardware friendly; however, they do not effectively leverage the depth of the SNN architectures (we show it empirically here), thus, are limited. In this work, we present an improved version of a conventional three-factor rule, for local learning in SNNs which effectively leverages depth – in the context of learning features hierarchically. Taking inspiration from the Back-propagation algorithm, we theoretically derive our improved, local, three-factor based learning method, named DALTON (Deep LocAl Learning via local WeighTs and SurrOgate-Derivative TraNsfer), which employs weights and surrogate-derivative transfer from the local layers. Along the lines of TFR, our proposed method DALTON is also amenable to the neuromorphic-hardware implementation. Through extensive experiments on static (MNIST, FMNIST, & CIFAR10) and event-based (N-MNIST, DVS128-Gesture, & DVS-CIFAR10) datasets, we show that our proposed local-learning method DALTON makes effective use of the depth in Convolutional SNNs, compared to the vanilla TFR implementation.