Workflow overhead analysis and optimizations

Abstract

The execution of scientific workflows often suffers from a variety of overheads in distributed environments. It is essential to identify the different overheads and to evaluate how optimization methods help reduce overheads and improve runtime performance. In this paper, we present an overhead analysis for a set of workflow runs on cloud and grid platforms. We present the overhead distributions and conclude that they satisfy an exponential or uniform distribution. We compare three methods to calculate the cumulative sum of the overheads based on how they overlap. In addition, we indicate how experimental parameters impact the overhead and thereby the overall workflow performance. We then show how popular optimization methods improve runtime performance by reducing some or all types of overheads.