Leveraging Prediction Confidence For Versatile Optimizations to CNNs

Abstract

Modern convolutional neural networks (CNNs) incur huge computational and energy overheads. In this paper, we propose two techniques for inferring the confidence in the correctness of a prediction in the early layers of a CNN. The first technique uses a statistical approach, whereas the second technique requires retraining. We argue that prediction confidence estimation can enable diverse optimizations to CNNs. We demonstrate two optimizations. First, we predict selected images in early layers. This is possible because in a dataset, many images are easy to predict and they can be predicted in the early layers of a CNN. This reduces the average computation count at the cost of accuracy and parameter count. Second, we propose predicting only selected images for which the prediction-confidence is high. This reduces the coverage; however, the accuracy on the images that are predicted is higher. Our results with VGG16 and ResNet50 CNNs on the Caltech256 dataset show that our techniques are effective. For example, for ResNet, our first technique reduces the accuracy from 71.6% to 69.8% while reducing the computations by 14%. Similarly, with the second technique, on reducing the coverage from 100% to 90%, the accuracy is increased from 71.6% to 75.6%. Keywords: computer vision, CNN, approximate computing, accuracy-coverage tradeoff, prediction confidence