Proximal optimization for resource allocation in distributed computing systems with data locality

Abstract

We consider resource allocation questions for computing infrastructures with multiple server instances. In particular, the joint optimization of active service capacity, load balancing between clusters of servers, and task scheduling at each cluster, under conditions of data locality which imply different service rates for different cluster locations.Building on previous work, we formulate a convex optimization problem, and use Lagrange duality to decompose it between the different decision variables. We include regularization terms from proximal methods to obtain continuous control laws for load balancing and scheduling, and optimize the remaining variables through primal-dual gradient dynamics. We prove convergence of the resulting control laws to the desired optimal points, and demonstrate its behavior by simulations.