Multi-Model Inference Acceleration on Embedded Multi-Core Processors

Abstract

The predominant resource efficient approaches that enable on-device inference include designing lightweight DNN architectures like MobileNets, SqueezeNets, compressing model using techniques such as network pruning, vector quantization, distillation, binarization. Recent research on using dynamic layer-wise partitioning and partial execution of CNN based model inference also make it possible to co-inference on memory and computation resource constrained devices. However, these approaches have their own bottleneck, lightweight DNN architectures and model compression usually compromise accuracy in order to deploy on resource constrained devices, dynamic model partitioning efficiency depends heavily on the network condition. This paper proposes an approach for multimodel inference acceleration on heterogeneous devices. The idea is to deploy multiple single object detection model instead of one heavy multiple object, this is because in most cases it only needs to detect one or two objects in one scenario and single object detection model weight could be lighter for the same resolution quality and require less resource. Moreover, in cloud-edge-device scenario, with the help of a scheduler policy, it is possible to gradually update models in need.