Decision Early-Exit: An Efficient Approach to Hasten Offloading in BranchyNets

Many works study partitioning and early exits in Deep Neural Networks (DNNs) to improve the inference time. Early exits allow the inference of samples in advance, based on the fact that some features are learned at DNNs’ initial layers. However, usage of early exits can slightly decrease performance. Partitioning enables the shallowest part of the model to reside at the edge while the deeper layers reside in the cloud. Deciding whether samples must be sent to the cloud at each early exit is time consuming, increasing the total inference time. Hence, reducing this time while maintaining the model performance is currently an open challenge. In this paper, we propose a Decision Early Exit (DEEx), implemented at the first early exit, aiming to reduce the total inference time by skipping unnecessary evaluations at early exits, which may not able to improve the model’s performance. To this end, the DEEx compares a predefined decision threshold with the prediction confidence level for each sample and decides whether the sample must be offloaded. We assess DEEx through a comparative analysis that investigates the influence of different values for the decision threshold on the inference time. Our results show that there is a cost benefit between the inference time and the threshold. Using DEEx in a simulated BranchyNet, we can reduce the inference time by around 20% while maintaining the same accuracy achieved when the samples are offloaded.