Inference framework supporting parallel execution across heterogeneous accelerators

Abstract

The growing demand for on-device deep learning inference, particularly in latency-sensitive applications, has driven the adoption of heterogeneous accelerators that incorporate GPUs, DSPs, and NPUs. While these accelerators offer improved energy efficiency, their heterogeneity introduces significant programming complexity due to reliance on vendor-specific APIs. Existing deep learning inference frameworks, such as LiteRT, provide high-level APIs and support multiple backend APIs. However, they lack the ability to exploit parallel execution across heterogeneous accelerators. This paper introduces a novel inference framework that overcomes this limitation. Our framework utilizes a batch inference API to enable parallel execution across heterogeneous accelerators. The framework schedules heterogeneous accelerators to process batched inputs concurrently. To address the challenge of integrating commercial NPU APIs that do not support LiteRT, we develop a portable hooking engine. Furthermore, the framework mitigates accuracy inconsistencies arising from diverse quantization methods by dynamically adjusting postprocessing parameters to balance accuracy and latency. The proposed framework minimizes both average turnaround time and postprocessing time. Experimental results on a Qualcomm SA8195 SoC with Mobilint and Hailo NPUs demonstrate significant performance improvements compared to existing inference frameworks.