Coordination of data movement with computation scheduling on a cluster

Abstract

We are looking at the problem of scheduling compute tasks on a cluster of servers. These tasks require files that reside on a remote archive, and may also be cached on some subset of the servers. A task can only be run on a server that has the files it requires. This introduces the problem of scheduling data movement in coordination with the scheduling of computation. Our goal is to maximize throughput while minimizing data movement. FIFO scheduling is not efficient in this situation due to its lack of awareness of the data movement required. We looked at two other strategies, called shortest job first and linear programming based optimization, and compared them under various configurations.