Re-Calibrating Network by Refining Initial Features Through Generative Gradient Regularization

Abstract

In the domain of Deep Neural Networks (DNNs), the deployment of regularization techniques is a common strategy for optimizing network performance. While these methods have been shown to be effective for optimization, they typically necessitate complete retraining of the network. We propose a training methodology that emphasizes on refining the features extracted from the initial layer of a DNN by regularizing the network with the help of a reference gradient. Our findings indicate that augmenting the gradients produced by the filters of the initial layer of a DNN, through the introduction of a reference gradient, leads to refined feature extraction and enhanced performance. We produce the reference gradient from the decoder of a generative network and subsequently encourage the target classifier network to adjust its weights to minimize discrepancies between the reference gradient and the gradient produced by the classifier network. The experiments show that implementing this method on a pre-trained network effectively re-calibrates the network and augments higher variance filters of the initial layer of the network, which helps produce refined features. Notably, this refinement in features translates to improved generalization and the proposed method also eliminates the necessity of total retraining of the target network. In empirical evaluation, we applied the proposed methodology to CIFAR, SVHN and ImageNet datasets, utilizing a range of network architectures. The results evidenced a performance gain of 1.66% for the CIFAR dataset using WideResNet, 1.22% for the SVHN dataset using PreResNet and 0.57% for the ImageNet dataset using ResNet.