Characterizing and Accelerating End-to-End EdgeAI Inference Systems for Object Detection Applications

Modern EdgeAI inference systems still have many cruciallimi-tations. In this paper, we holistically consider implications and optimizations of EdgeAI inference systems for object detection applications in efficiency and accuracy. We summarize three in-trinsic limitations of current-generation EdgeAI inference systems based on our observations (i.e., less compute capabilities, restrictions of operations, and accuracy loss due to numerical precision). Then we propose three approaches to improve end-to-end performance and prediction accuracy: 1) Utilizing parallel computing designs and methods to solve computational bottlenecks; 2) Ap-plying domain-specific optimizations to mostly eliminate accuracy loss; 3) Using higher-quality input data to saturate the processors and accelerators. We also provide five recommendations for end-to-end EdgeAI solution deployments, which are usually neglected by EdgeAI users. In particular, we deploy and optimize two real object detection applications (2D and 3D) on two EdgeAI inference systems (NovuTensor and Nvidia Xavier) with widely used datasets (i.e., MS-COCO, PASCAL-VOC, and KITTI). The results show that runtime performance can be accelerated by up to 2X on NovuTen-sor and the mean average precision (mAP) can be increased by 46% through applying our proposed methods.