Optimizing Server Allocation for Multi-Tenant GPU Inference With Dynamic Batching: A Queueing Theoretic Approach

Abstract

Efficient GPU resource management is essential for multi-tenant deep neural network (DNN) inference platforms, where dynamic batching is widely used to improve throughput and energy efficiency. However, batching introduces a trade-off between latency and batch size, particularly when diverse applications compete for a limited server pool. Existing measurement-based tuning approaches often require re-profiling system performance under varying arrival rates, which renders them impractical for dynamic production environments. This paper introduces a queueing-theoretic framework for application-aware server allocation under energy constraints. We model each application’s server using a realistic dynamic batching policy and Erlang interarrival processes, arising from Poisson job arrivals and round-robin job dispatching. We derive closed-form approximations for expected batch size and job latency. These approximations enable a convex optimization that minimizes the maximum average latency across applications while satisfying batch-dependent energy efficiency constraints. The proposed method eliminates the need for exhaustive simulation or empirical profiling across different arrival rates, making it well-suited for dynamic, real-time server provisioning. Experiments on a real GPU server with ResNet-50 and RetinaNet demonstrate that our approach closely approximates optimal allocation strategies while significantly reducing computational overhead. The framework supports both server-based and serverless deployments and is readily applicable to modern DNN inference platforms.