The X-flex cross-platform scheduler: who's the fairest of them all?

Abstract

We introduce the X-Flex cross-platform scheduler. X-Flex is intended as an alternative to the Dominant Resource Fairness (DRF) scheduler currently employed by both YARN and Mesos. There are multiple design differences between X-Flex and DRF. For one thing, DRF is based on an instantaneous notion of fairness, while X-Flex monitors instantaneous fairness in order to take a long-term view. The definition of instantaneous fairness itself is different among the two schedulers. Furthermore, the packing of containers into processing nodes in DRF is done online, while in X-Flex it is performed offline in order to improve packing quality. Finally, DRF is essentially an extension to multiple dimensions of the Fair MapReduce scheduler. As such it makes scheduling decisions at a very low level. X-Flex, on the other hand, takes the perspective that some frameworks have sufficient structure to make higher level scheduling decisions. So X-Flex allows this, and also gives platforms a great deal of autonomy over the degree of sharing they will permit with other platforms. We describe the technical details of X-Flex and provide experiments to show its excellent performance.