Brokering Solution for Science Gateways Using Multiple Distributed Computing Infrastructures

Abstract

The ever growing number of computation-intensive applications calls for the interoperation of distributed infrastructures such as Clouds, Grids and private clusters. The European SHIWA and ER-flow projects have been initiated to enable the combination of heterogeneous scientific workflows, and to execute them in a large-scale system consisting of multiple Distributed Computing Infrastructures including Grids and Clouds. One of the resource management challenges of these projects called parameter study job scheduling. A parameter study job of a workflow has a large number of input files to be consumed by independent job instances. In this paper we propose a meta-brokering solution for these workflows to be applied in science gateways. To cope with the high uncertainty and unpredictable load of the utilized infrastructures, we rely on resource priority services. These tools are capable of determining and dynamically updating priorities of the available infrastructures to be selected for job instances.