Work-in-Progress: BPNet: Branch-pruned Conditional Neural Network for Systematic Time-accuracy Tradeoff in DNN Inference

Abstract

Recently, there have been attempts to execute the neural network conditionally with auxiliary classifiers allowing early termination depending on the difficulty of the input, which can reduce the execution time or energy consumption without any or with negligible accuracy decrease. However, these studies do not consider how many or where the auxiliary classifiers, or branches, should be added in a systematic fashion. In this paper, we propose Branch-pruned Conditional Neural Network (BPNet) and its methodology in which the time-accuracy tradeoff for the conditional neural network can be found systematically. We applied BPNet to SqueezeNet, ResNet-20, and VGG-16 with CIFAR-10 and 100. BPNet achieves on average 2.0x of speedups without any accuracy drop on average compared to the base network.