Real-Time Inference Platform for Object Detection on Edge Device

Abstract

Deep Neural Networks (DNNs) that perform object detection, such as autonomous driving, facial recognition, and medical healthcare, have received great attention recently. Due to the large amounts of data used in object detection DNNs, a cloud computing system for AI with centralized computing power and storage capacity has been used. However, with the increasing number of edge devices in the IoT trend and the growing amount of data, cloud-based AI encounters a challenge of network latency in processing real-time inference. In this paper, we propose a platform consisting of an edge device with a DNN inference accelerator and an optimized network to address the latency issues and achieve real-time inference of DNNs. The proposed platform adopts SqueezeNet, which is suitable for mobile devices due to its smaller network size than other DNNs. Post Training Quantization compresses the pre-trained SqueezeNet model size without accuracy loss. With the compressed network, an xczu3eg chip-based MPSoC board that includes an AI accelerator is used as the edge device. To further improve inference throughput, multi-threading is also used to reduce the latency between the Processing System(PS) and Programmable Logic(PL). Through the proposed platform, we achieve a 55 frame-per-second(fps) throughput, which is a sufficient real-time object detection inference performance.