Deep Residual Networks with Exponential Linear Unit

The depth of convolutional neural networks is a crucial ingredient for reduction in test errors on benchmarks like ImageNet and COCO. However, training a neural network becomes difficult with increasing depth. Problems like vanishing gradient and diminishing feature reuse are quite trivial in very deep convolutional neural networks. The notable recent contributions towards solving these problems and simplifying the training of very deep models are Residual and Highway Networks. These networks allow earlier representations (from the input or those learned in earlier layers) to flow unimpededly to later layers through skip connections. Such very deep models with hundreds or more layers have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO. In this paper, we propose to replace the combination of ReLU and Batch Normalization with Exponential Linear Unit (ELU) in Residual Networks. Our experiments show that this not only speeds up the learning behavior in Residual Networks, but also improves the classification performance as the depth increases. Our model increases the accuracy on datasets like CIFAR-10 and CIFAR-100 by a significant margin.