Enable Pipeline Processing of DNN Co-inference Tasks In the Mobile-Edge Cloud

Abstract

Deep Neural Network (DNN) based artificial intelligence help driving the great development of mobile Internet. However, the hardware of a mobile device may not be sufficiently to meet the computational requirements of a DNN inference task. Fortunately, computation offloading to the network edge can mitigate part of computation pressure for mobile devices. In this case, DNN computation in mobile devices can be accelerated by an edge-assistance collaborative inference scheme. Since co-inference tasks with multiple processing stages may continuously arrive at mobile devices, only considering one DNN-based task for acceleration is not practical. To solve the above challenge effectively, we formulate the problem of multiple co-inference tasks acceleration as a pipeline execution model. Based on the model, we design a fine-grained optimizer, which integrates model partition, model early-exit and intermediate data compression, to achieve tradeoff between accuracy and latency. Considering computational characteristics of a pipeline, the goal of the optimizer is designed to ensure the pipeline system's inference rate and single task execution performance. We implement the system prototype and do benchmark tests under a real-life testbed and the results prove the effectiveness of the optimizer.