ML-ACE: Machine Learning Admission Control at the Edge

Abstract

ML inference has become an increasingly important workload for low-power, near-data edge computing platforms. There is a large existing body of work on how to optimize a trained model for inference on a resource-constrained device, however much of the work does not consider optimizations in how the model will be used by clients in the system. In this space, inference servers emerged to provide a client-server paradigm for inference, offering portable, practical client libraries for users of ML systems. These servers handle batching of requests, runtime optimizations, and placement of multiple replicas of models on CPU/GPU to maximize inference efficiency. Unlike the data center, much infrastructure at the edge lacks the ease in ability to recruit new machines to scale out these servers to meet increasing request demand. Because of this, efficient scheduling of models on these edge platforms is critical. This work presents ML-ACE, a system to systematically schedule ML inference on resource-constrained edge computing platforms. ML-ACE extends the existing client-server paradigm for inference serving by providing admission control, preventing user inference requests from over-saturating system resources.