Beehive: Decentralised High-Frequency Small Tasks Scheduling in Large Clusters

Data centers struggle with growing cluster sizes and rising submissions of short-lived, high-frequency tasks that cause performance bottlenecks in task scheduling. Existing centralized and distributed scheduling systems fall short in meeting performance requirements due to computational overload on the scheduler, cluster state management overhead, and scheduling conflicts. To address these challenges, this article introduces Beehive, a novel lightweight decentralized scheduling framework. In Beehive, each cluster node can schedule tasks within its local neighborhood, effectively reducing resource management overhead and scheduling conflicts. Moreover, all nodes are interconnected in a small-world network, an efficient structure that allows tasks to access resources across the entire cluster through global routing. This lightweight design enables Beehive to scale efficiently, supporting over 10,000 nodes and up to 80,000 task submissions per second without causing single-node scheduling bottlenecks. Experimental results demonstrate that Beehive significantly reduces scheduling latency. Specifically, 99% of tasks are scheduled within 100 milliseconds, and scheduling throughput can increase linearly with the number of nodes. Compared to existing centralized and distributed scheduling frameworks, Beehive substantially alleviates scheduling bottlenecks, particularly for high-frequency, short-lived tasks.