A Highly Scalable Decentralized Scheduler of Tasks with Deadlines

Abstract

Scheduling of tasks in distributed environments, like cloud and grid computing platforms, using deadlines to provide quality of service is a challenging problem. The few existing proposals suffer from scalability limitations, because they try to manage full knowledge of the system state. To our knowledge, there is no implementation yet that reaches scales of a hundred thousand nodes. In this paper, we present a fully decentralized scheduler, that aggregates information about the availability of the execution nodes throughout the network and uses it to allocate tasks to those nodes that are able to finish them in time. Through simulation, we show that our scheduler is able to operate on different scenarios, from many-task applications in cloud computing sites to volunteer computing projects. Simulations on networks of up to a hundred thousand nodes show very competitive performance, reaching allocation times of under a second and very low overhead in low latency gigabit networks.