DBM: Delay-sensitive Buffering Mechanism for DNN Offloading Services

Abstract

DNN offloading has become an important supporting technology for edge intelligence. However, most of the existing works do not consider thread scheduling, which can achieve the parallelism of multiple threads in the practical distributed DNN inference system. To address this issue, we discuss the thread scheduling of the computing units participating in offloading in this paper, considering a single-core Central Processing Unit (CPU) and the Round Robin Scheduling (RRS). We deduce the relationship between the blocking of DNN inference-related threads and the Average Task Delay (ATD) and prove that an appropriate buffer setting can reduce blocking times. Theoretical analysis verifies that the buffering mechanism (DBM) can reduce the ATD significantly, and experimental results demonstrate that the DBM-improved DNN offloading can achieve a delay reduction of 14%-71%.