Accelerating Parameter Sweep Workflows by Utilizing Ad-hoc Network Computing Resources: An Ecological Example

Abstract

Making use of distributed execution within scientific workflows is a growing and promising methodology to achieve better execution performance. We have implemented a distributed execution framework in the Kepler scientific workflow environment, called Master-Slave Distribution, to distribute sub-workflows to a common distributed environment, namely ad-hoc network computing resources. For a typical parameter sweep workflow, this architecture can realize concurrent independent sub-workflow executions with minimal user configuration, allowing large gains in productivity with little of the typical overhead associated with learning distributed computing systems. We explain details of the Master-Slave architecture and demonstrate its usability and time efficiency by a use case in the theoretical ecology domain. We also discuss the capabilities of this architecture under different computational domains in Kepler.