A Genetic Programming Hyper-Heuristic Approach to Design High-Level Heuristics for Dynamic Workflow Scheduling in Cloud

Workflow scheduling in the cloud is the process of allocating tasks to scarce cloud resources, with an optimal goal. This is often achieved by adopting an effective scheduling heuristic. Workflow scheduling in cloud is challenging due to the dynamic nature of the cloud, often existing works focus on static workflows, ignoring this challenge. Existing heuristics, such as MINMIN, focus mainly on one specific aspect of the scheduling problem. High-level heuristics are heuristics constructed from existing man-made heuristics. In this paper, we introduce a new and more realistic workflow scheduling problem that considers different kinds of workflows, cloud resources and high-level heuristics. We propose a High-Level Heuristic Dynamic Workflow Scheduling Genetic Programming (HLH-DSGP) algorithm to automatically design high-level heuristics for workflow scheduling to minimise the response time of dynamically arriving task in a workflow. Our proposed HLH-DSGP can work consistently well regardless of the size and pattern of workflows, or number of available cloud resources. It is evaluated using a popular benchmark dataset using the popular WorkflowSim simulator. Our experiments show that with high-level scheduling heuristics, designed by HLH-DSGP, we can jointly use several well-known heuristics to achieve a desirable balance among multiple aspects of the scheduling problem collectively, hence improving the scheduling performance.