Optimizing the Data-Traffic of Centrally Coordinated Scientific Workflow Systems

Abstract

Scientific workflow systems facilitate scientific experiments by integrating and coordinating geographically distributed data and algorithmic services in a loosely coupled manner. Most scientific workflow-engines use centralized coordination as the choice of approach for executing workflows, requiring the coordinator (i.e., workflow-engine) to send and receive all input and output data of component services. Such indirect data communication between the component services increases the data-traffic of the coordinator and weakens the performance of the workflow. To optimize this, we propose an approach where data-flow is dynamically delegated from the coordinator to the component services, with direct transportation of data between the component services.