Complexity Reduction: Local Activity Ranking by Resource Entropy for QoS-Aware Cloud Scheduling

The principle of local activity originated from electronic circuits, but can easily translate into other non-electrical homogeneous/heterogeneous media. Cloud resource is an example of a locally-active device, which is the origin of complexity in cloud scheduling system. However, most of the researchers implicitly assume the cloud resource to be locally passive when constructing new scheduling strategies. As a result, their research solutions perform poorly in the complex cloud environment. In this paper, we first study several complexity factors caused by the locally-active cloud resource. And then we extended the ”Local Activity Principle” concept with a quantitative measurement based on Entropy Theory. Furthermore, we classify the scheduling system into ”Order” or ”Chaos” state with simulating complexity in the cloud. Finally, we propose a new approach to controlling the chaos based on resource's Local Activity Ranking for QoS-aware cloud scheduling and implement such idea in Spark. Experiments demonstrate that our approach outperforms the native Spark Fair Scheduler with server cost reduced by 23%, average response time improved by 15% - 20% and standard deviation of response time minimized by 30% - 45%.