Calibrated gradient descent of convolutional neural networks for embodied visual recognition

Embodied visual computing seeks to learn from the real world, which requires an efficient machine learning methods. In conventional stochastic gradient descent (SGD) and its variants, the gradient estimators are expensive to be computed in many scenarios. This paper introduces a calibrated gradient descent (CGD) algorithm for efficient deep neural network optimization. A theorem is developed to prove that an unbiased estimator for network parameters can be obtained in a probabilistic way based on the Lipschitz hypothesis. We implement our CGD algorithm based on widely-used SGD and ADAM optimizers. We achieve a generic gradient calibration layer ($GClayer$) which can be used to improve the performance of convolutional neural networks (C-CNNs). Our $GClayer$ only introduces extra parameters during training process, but not affect the efficiency of the inference process. Our method is generic and effective to optimize both CNNs and also quantized neural networks (C-QNNs). Extensive experimental results demonstrate that our method can achieve the state-of-the-art performance for a variety of tasks. For example, our 1-bit Faster-RCNN achieved by C-QNN obtains 20.5% mAP on COCO, leading to a new state-of-the-art performance. This work brings new insights for developing more efficient optimizers and analyzing the back-propagation algorithm.