A Web Service-Enabled Distributed Workflow System for Scientific Data Processing

Abstract

This paper presents the design and implementation of a distributed data-driven workflow system on top of the TeraGrid infrastructure. The workflow system is based on a data management architecture that provides easy access to scientific data collections via the TeraGrid network. The workflow system allows researchers to construct scientific workflows for data discovery, access, transformation, and analysis. The system leverages JOpera, an open-source workflow engine and visual composer, as well as a set of Web service-based data and computation modules. To demonstrate its effectiveness, we create an end-to-end climate simulation data analysis workflow that connects the data management architecture to TeraGrid computation resources. We also develop a workflow monitoring service to keep track of distributed workflow execution