Deadline Distribution Strategies for Scientific Workflow Scheduling in Commercial Clouds

Abstract

Commercial clouds have become a viable platform for performing a significant range of large scale scientific analyses – due to the offerings of elasticity, specialist hardware, software infrastructure and pay-as-you-go cost model. Such clouds represent a low upfront capital cost alternative to the use of dedicated eScience infrastructure. However, there are still significant technical hurdles associated with obtaining the best performance for the cost - it is easy to provision commercial clouds inefficiently resulting in great and potentially unanticipated expense. In this paper we introduce a new heuristic scheduling algorithm Deadline Distribution Ratio (DDR) to address the workflow scheduling problem with the objectives of minimizing the cost of Cloud computing resources while satisfying a given deadline. Within this context, we also investigate a range of different deadline distribution strategies and their effect on the overall scheduling performance. We then compare the DDR algorithm against three other published algorithms, using five different scientific workflows generated using the pegasus workflow generator, on a CloudSim simulation that implements a pricing model based on AWS. In general, the DDR algorithm returns the lowest costs across the majority of deadlines and workflows, while maintaining a high scheduling success rate.