Scheduler for data-intensive workflows in public clouds

Abstract

Data-intensive workflows can require the use of intermediary data storage in the order of terabytes. Thus, planning the execution of such workflows in the cloud considering only processing demand, regardless its data storage needs, leads to performance decrease and potential increase in costs. In this paper, we present an integer linear program scheduler that considers disk storage scheduling besides the task scheduling based on processor time. The proposed scheduler aims to achieve the lowest economic cost while serving a deadline set by the user. The results show that the scheduler can find good schedules in situations where the disk size of rented virtual machines is a limiting factor.